Re: svn commit: r1784228 - in /httpd/httpd/trunk: docs/manual/mod/mod_proxy_hcheck.xml modules/proxy/mod_proxy_hcheck.c
On Fri, Feb 24, 2017 at 5:49 AM, wrote: > Author: jim > Date: Fri Feb 24 04:49:38 2017 > New Revision: 1784228 > > URL: http://svn.apache.org/viewvc?rev=1784228&view=rev > Log: > Instead of thread-pools for each server/context, use a server-wide > top-level threadpool, which is more efficient. > > Modified: > httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c > > Modified: httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c > URL: > http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c?rev=1784228&r1=1784227&r2=1784228&view=diff > == > --- httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c (original) > +++ httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c Fri Feb 24 04:49:38 > 2017 > > @@ -86,13 +84,15 @@ static void *hc_create_config(apr_pool_t > ctx->templates = apr_array_make(p, 10, sizeof(hc_template_t)); > ctx->conditions = apr_table_make(p, 10); > ctx->hcworkers = apr_hash_make(p); > -ctx->tpsize = HC_THREADPOOL_SIZE; > ctx->s = s; > > return ctx; > } > > static ap_watchdog_t *watchdog; > +static apr_thread_pool_t *hctp = NULL; > +static int tpsize = HC_THREADPOOL_SIZE; Maybe init 'tpsize' in pre_config for --enable-proxy_check=static restarts? Regards, Yann.
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 10:06 PM, Daniel Lescohier wrote: > Why would high-order memory allocations be a problem in userspace code, > which is using virtual memory? I thought high-order allocations is a big > problem in kernel space, which has to deal with physical pages. Well, both in kernel or user space, the difficulty is finding large contiguous memory. With virtual memory (admittedly virtually larger than physical memory), it needs more "active" regions to fail, but still it can fail if many heterogeneous chunks are to be mapped at a time, the OOM killer will do its job. It depends on how closed to the resident memory limit you are of course (it won't happen if some memory can be compressed or swapped), but large chunks are no better with lot of RAM either. > > But when you write to a socket, doesn't the kernel scatter the userspace > buffer into multiple SKBs? SKBs on order-0 pages allocated by the kernel? Right, in Linux network stack (or drivers) is mainly using (or is moving to) order 0 or 1 chunks (with scatterlists when needed). But this is where httpd's job end, we talk about before this here :) >From the other message: On Wed, Feb 22, 2017 at 8:55 PM, Daniel Lescohier wrote: > > IOW: > read():Three copies: copy from filesystem cache to httpd read() buffer to > encrypted-data buffer to kernel socket buffer. Not really, "copy from filesystem cache to httpd read() buffer" is likely mapping to userspace, so no copy (on read) here. > mmap(): Two copies: filesystem page already mapped into httpd, so just copy > from filesystem (cached) page to encrypted-data buffer to kernel socket > buffer. So, as you said earlier the "write to socket" isn't a copy either, hence both read() and mmap() implementations could work with a single copy when mod_ssl is involved (this is more than a copy but you counted it above so), and no copy at all without it. Regards, Yann.
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 8:50 PM, Jacob Champion wrote: > On 02/22/2017 02:16 PM, Niklas Edmundsson wrote: > > I don't think s_server is particularly optimized for performance anyway. > > Oh, and just to complete my local testing table: > > - test server, writing from memory: 1.2 GiB/s > - test server, mmap() from disk: 1.1 GiB/s > - test server, 64K read()s from disk: 1.0 GiB/s > - httpd trunk with `EnableMMAP on` and serving from disk: 850 MiB/s > - httpd trunk with 'EnableMMAP off': 580 MiB/s > - httpd trunk with my no-mmap-64K-block file bucket: 810 MiB/s > > My test server's read() implementation is a really naive "block on read, > then block on write, repeat" loop, so there's probably some improvement to > be had there, but this is enough proof in my mind that there are major gains > to be made regardless. Does no-mmap-2MB-block beats MMap on? > >> Going off on a tangent here: >> >> For those of you who actually know how the ssl stuff really works, is it >> possible to get multiple threads involved in doing the encryption, or do >> you need the results from the previous block in order to do the next >> one? > > I'm not a cryptographer, but I think how parallelizable it is depends on the > ciphersuite in use. Like you say, some ciphersuites require one block to be > fed into the next as an input; others don't. Yes, and the cost of scheduling threads for non dedicated cypto device is not worth it I think. But mainly there is not only one stream involved in a typical HTTP server, so probably multiple simulteneous connections already saturate the AES-NI...
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 7:16 PM, Jacob Champion wrote: > On 02/23/2017 08:34 AM, Yann Ylavic wrote: >> Actually I'm not very pleased with this solution (or the final one >> that would make this size open / configurable). >> The issue is potentially the huge (order big-n) allocations which >> finally may hurt the system (fragmentation, OOM...). > > Power users can break the system, and this is a power tool, right? It's not about power users, I don't think we can recommend anyone to use 4MB buffers even if they (seem to) have RAM for it. > And we > have HugeTLB kernels and filesystems to play with, with 2MB and bigger > pages... Making all these assumptions for 90% of users is perfect for the > out-of-the-box experience, but hardcoding them so that no one can fix broken > assumptions seems Bad. I'm not talking about hardcoding anything, nor reading or sendind hard limited sizes on filesystem/sockets. I'm proposing that the configuration relates to how much we "coalesce" data on output, and all buffers' reuses (though each of fixed size) should follow the needs. > > (And don't get me wrong, I think applying vectored I/O to the brigade would > be a great thing to try out and benchmark. I just think it's a long-term and > heavily architectural fix, when a short-term change to get rid of some > #defined constants could have immediate benefits.) Of course, apr_bucket_file_set_read_size() is a quick patch (I dispute it for the general case, but it may be useful, or not, for the 16K SSL case, let's see), and so is another for configuring core_output_filter constants, but we don't need them for testing right? > >> I've no idea how much it costs to have 8K vs 16K records, though. >> Maybe in the mod_ssl case we'd want 16K buffers, still reasonable? > > > We can't/shouldn't hardcode this especially. People who want maximum > throughput may want nice big records, but IIRC users who want progressive > rendering need smaller records so that they don't have to wait as long for > the first decrypted chunk. It needs to be tunable, possibly per-location. Well, the limit is 16K at the TLS level. Regards, Yann.
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 7:15 PM, Niklas Edmundsson wrote: > On Thu, 23 Feb 2017, Yann Ylavic wrote: > >>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just >>> defines a new buffer size for use with the file bucket. It's a little >>> less >>> than 64K, I assume to make room for an allocation header: >>> >>> #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) >> >> >> Actually I'm not very pleased with this solution (or the final one >> that would make this size open / configurable). >> The issue is potentially the huge (order big-n) allocations which >> finally may hurt the system (fragmentation, OOM...). > > > Is this a real or theoretical problem? Both. Fragmentation is a hard issue, but a constant is: the more you ask for big allocs, the more likely you won't be serviced one day or another (or another task will be killed for that, until yours). Long living (or closed to memory limits) systems suffer from this, no matter what allocator, small and large allocations fragment the memory (httpd is likely not the only program on the system), the only "remedy" is small order allocations (2^order pages, a "sane" order being lower than 2, hence order-1 on a system with 4K pages is 8K bytes). > > Our large-file cache module does 128k allocs to get a sane block size when > copying files to the cache. The only potential drawback we noticed was httpd > processes becoming bloated due to the default MaxMemFree 2048, so we're > running with MaxMemFree 256 now. With MaxMemFree 256 (KB per allocator), each APR allocator reclaims but mainly returns a lot more memory chunks to the system's allocator, which does a better job to in recycling many differently sized chunks than APR's (which is pretty basic in this regard, it's role is more about quickly recycling common sizes). With MaxMemFree 2048 (2MB), there is more builtin handling of "exotic" chunks, which may be painful. That might be something else, though... > > Granted, doing alloc/free for all outgoing data means way more alloc/free:s, > so we might just miss the issues because cache fills aren't as common. That's why reuse of common and reasonably sized chunks in the APR allocator can help. > > However, for large file performance I really don't buy into the notion that > it's a good idea to break everything into tiny puny blocks. The potential > for wasting CPU cycles on this micro-management is rather big... I don't think that a readv/writev of 16 iovecs of 8KB is (noticeably) slower than read/write of contiguous 128K, both might well end up in a scaterlist for the kernel/hardware. > > I do find iovecs useful, it the small blocks that gets me into skeptic > mode... Small blocks is not for networking, it's for internal use only. And remember that TLS records are 16K max anyway, give 128KB to openssl's SSL_write it will output 8 chunks of 16KB. > > > Kinda related: We also have the support for larger page sizes with modern > CPUs. Has anyone investigated if it makes sense allocating memory pools in > chunks that fit those large pages? I think PPC64 have 64K pages already. APR pools are already based on the page size IIRC. Regards, Yann.
Re: httpd 2.4.25, mpm_event, ssl: segfaults
Why would high-order memory allocations be a problem in userspace code, which is using virtual memory? I thought high-order allocations is a big problem in kernel space, which has to deal with physical pages. But when you write to a socket, doesn't the kernel scatter the userspace buffer into multiple SKBs? SKBs on order-0 pages allocated by the kernel? On Thu, Feb 23, 2017 at 1:16 PM, Jacob Champion wrote: > On 02/23/2017 08:34 AM, Yann Ylavic wrote: > > Actually I'm not very pleased with this solution (or the final one > > that would make this size open / configurable). > > The issue is potentially the huge (order big-n) allocations which > > finally may hurt the system (fragmentation, OOM...). > > Power users can break the system, and this is a power tool, right? And we > have HugeTLB kernels and filesystems to play with, with 2MB and bigger > pages... Making all these assumptions for 90% of users is perfect for the > out-of-the-box experience, but hardcoding them so that no one can fix > broken assumptions seems Bad. > > (And don't get me wrong, I think applying vectored I/O to the brigade > would be a great thing to try out and benchmark. I just think it's a > long-term and heavily architectural fix, when a short-term change to get > rid of some #defined constants could have immediate benefits.) > > I've no idea how much it costs to have 8K vs 16K records, though. >> Maybe in the mod_ssl case we'd want 16K buffers, still reasonable? >> > > We can't/shouldn't hardcode this especially. People who want maximum > throughput may want nice big records, but IIRC users who want progressive > rendering need smaller records so that they don't have to wait as long for > the first decrypted chunk. It needs to be tunable, possibly per-location. > > --Jacob >
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On 02/22/2017 02:16 PM, Niklas Edmundsson wrote: Any joy with something simpler like gprof? (Caveat: haven't used it in ages to I don't know if its even applicable nowadays). Well, if I had thought about it a little more, I would have remembered that instrumenting profilers don't profile syscalls very well, and they especially mess with I/O timing. Valgrind was completely inconclusive on the read() vs. mmap() front. :( (...except that it showed that a good 25% of my test server's CPU time was spent inside OpenSSL in a memcpy(). Interesting...) So httpd isn't beat by the naive openssl s_server approach at least ;-) I don't think s_server is particularly optimized for performance anyway. Oh, and just to complete my local testing table: - test server, writing from memory: 1.2 GiB/s - test server, mmap() from disk: 1.1 GiB/s - test server, 64K read()s from disk: 1.0 GiB/s - httpd trunk with `EnableMMAP on` and serving from disk: 850 MiB/s - httpd trunk with 'EnableMMAP off': 580 MiB/s - httpd trunk with my no-mmap-64K-block file bucket: 810 MiB/s My test server's read() implementation is a really naive "block on read, then block on write, repeat" loop, so there's probably some improvement to be had there, but this is enough proof in my mind that there are major gains to be made regardless. Going off on a tangent here: For those of you who actually know how the ssl stuff really works, is it possible to get multiple threads involved in doing the encryption, or do you need the results from the previous block in order to do the next one? I'm not a cryptographer, but I think how parallelizable it is depends on the ciphersuite in use. Like you say, some ciphersuites require one block to be fed into the next as an input; others don't. --Jacob
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On 02/23/2017 08:34 AM, Yann Ylavic wrote: > Actually I'm not very pleased with this solution (or the final one > that would make this size open / configurable). > The issue is potentially the huge (order big-n) allocations which > finally may hurt the system (fragmentation, OOM...). Power users can break the system, and this is a power tool, right? And we have HugeTLB kernels and filesystems to play with, with 2MB and bigger pages... Making all these assumptions for 90% of users is perfect for the out-of-the-box experience, but hardcoding them so that no one can fix broken assumptions seems Bad. (And don't get me wrong, I think applying vectored I/O to the brigade would be a great thing to try out and benchmark. I just think it's a long-term and heavily architectural fix, when a short-term change to get rid of some #defined constants could have immediate benefits.) I've no idea how much it costs to have 8K vs 16K records, though. Maybe in the mod_ssl case we'd want 16K buffers, still reasonable? We can't/shouldn't hardcode this especially. People who want maximum throughput may want nice big records, but IIRC users who want progressive rendering need smaller records so that they don't have to wait as long for the first decrypted chunk. It needs to be tunable, possibly per-location. --Jacob
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, 23 Feb 2017, Yann Ylavic wrote: Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just defines a new buffer size for use with the file bucket. It's a little less than 64K, I assume to make room for an allocation header: #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) Actually I'm not very pleased with this solution (or the final one that would make this size open / configurable). The issue is potentially the huge (order big-n) allocations which finally may hurt the system (fragmentation, OOM...). Is this a real or theoretical problem? Our large-file cache module does 128k allocs to get a sane block size when copying files to the cache. The only potential drawback we noticed was httpd processes becoming bloated due to the default MaxMemFree 2048, so we're running with MaxMemFree 256 now. I don't know if things got much better, but it isn't breaking anything either... Granted, doing alloc/free for all outgoing data means way more alloc/free:s, so we might just miss the issues because cache fills aren't as common. However, for large file performance I really don't buy into the notion that it's a good idea to break everything into tiny puny blocks. The potential for wasting CPU cycles on this micro-management is rather big... I can see it working for a small-file workload where files aren't much bigger than tens of kB anyway, but not so much for large-file delivery. A prudent way forward might be to investigate what impact different block sizes have wrt ssl/https first. As networking speeds go up it is kind of expected that block sizes needs to go up as well, especially as per-core clock frequency isn't increasing much (it's been at 2-ish GHz base frequency for server CPUs the last ten years now?) and we're relying more and more on various offload mechanisms in CPUs/NICs etc to get us from 1 Gbps to 10 Gbps to 100 Gbps ... I do find iovecs useful, it the small blocks that gets me into skeptic mode... Kinda related: We also have the support for larger page sizes with modern CPUs. Has anyone investigated if it makes sense allocating memory pools in chunks that fit those large pages? /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se --- You need not worry about your future =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 5:34 PM, Yann Ylavic wrote: > On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing > wrote: >> >>> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : >>> >>> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion >>> wrote: On 02/22/2017 12:00 AM, Stefan Eissing wrote: > > Just so I do not misunderstand: > > you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you > are testing? Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable the mmap-optimization by default while still improving overall performance for most users. Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just defines a new buffer size for use with the file bucket. It's a little less than 64K, I assume to make room for an allocation header: #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) >>> >>> Actually I'm not very pleased with this solution (or the final one >>> that would make this size open / configurable). >>> The issue is potentially the huge (order big-n) allocations which >>> finally may hurt the system (fragmentation, OOM...). >>> >>> So I'm thinking of another way to achieve the same with the current >>> APR_BUCKET_BUFF_SIZE (2 pages) per alloc. >>> >>> The idea is to have a new apr_allocator_allocv() function which would >>> fill an iovec with what's available in the allocator's freelist (i.e. >>> spare apr_memnodes) of at least the given min_size bytes (possibly a >>> max too but I don't see the need for now) and up to the size of the >>> given iovec. >> >> Interesting. Not only for pure files maybe. >> >> It would be great if there'd be a SSL_writev()... > > Indeed, openssl would fully fill the TLS records. > >> but until there is, the TLS case will either turn every iovec into >> its own TLS record > > Yes, but it's more a client issue to work with these records finally > (because from there all is networking only, and coalescing will happen > in the core output filter from a socket POV). > > Anyway mod_proxy(s) could be the client, so... > > I've no idea how much it costs to have 8K vs 16K records, though. > Maybe in the mod_ssl case we'd want 16K buffers, still reasonable? > >> or one needs another copy before that. This last strategy is used by >> mod_http2. Since there are 9 header bytes per frame, copying data >> into a right sized buffer gives better performance. So, it would be >> nice to read n bytes from a bucket brigade and get iovecs back. >> Which, as I understand it, you propose? > > I didn't think of apr_bucket_readv(), more focused on > file_bucket_read() to do this internally/transparently, but once file > buckets can do that I think we can generalize the concept, at worse > filling only iovec[0] when it's ENOTIMPL and/or makes no sense... > > That'd help the mod_ssl case with something like > apr_bucket_readv(min_size=16K), I'll try to think of it once/if I can > have a more simple to do file_bucket_read() only working ;) Hm no, this needs to happen on the producer side, not at final apr_bucket_read(). So for the mod_ssl case I think we could have a simple (new) apr_bucket_alloc_set_size(16K) in the first place if it helps more than hurts. As for your question about "iovecs back" (which I finally didn't answer), all happens at the bucket alloc level, when buckets are deleted.
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing wrote: > >> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : >> >> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: >>> On 02/22/2017 12:00 AM, Stefan Eissing wrote: Just so I do not misunderstand: you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you are testing? >>> >>> >>> Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable >>> the mmap-optimization by default while still improving overall performance >>> for most users. >>> >>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just >>> defines a new buffer size for use with the file bucket. It's a little less >>> than 64K, I assume to make room for an allocation header: >>> >>>#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) >> >> Actually I'm not very pleased with this solution (or the final one >> that would make this size open / configurable). >> The issue is potentially the huge (order big-n) allocations which >> finally may hurt the system (fragmentation, OOM...). >> >> So I'm thinking of another way to achieve the same with the current >> APR_BUCKET_BUFF_SIZE (2 pages) per alloc. >> >> The idea is to have a new apr_allocator_allocv() function which would >> fill an iovec with what's available in the allocator's freelist (i.e. >> spare apr_memnodes) of at least the given min_size bytes (possibly a >> max too but I don't see the need for now) and up to the size of the >> given iovec. > > Interesting. Not only for pure files maybe. > > It would be great if there'd be a SSL_writev()... Indeed, openssl would fully fill the TLS records. > but until there is, the TLS case will either turn every iovec into > its own TLS record Yes, but it's more a client issue to work with these records finally (because from there all is networking only, and coalescing will happen in the core output filter from a socket POV). Anyway mod_proxy(s) could be the client, so... I've no idea how much it costs to have 8K vs 16K records, though. Maybe in the mod_ssl case we'd want 16K buffers, still reasonable? > or one needs another copy before that. This last strategy is used by > mod_http2. Since there are 9 header bytes per frame, copying data > into a right sized buffer gives better performance. So, it would be > nice to read n bytes from a bucket brigade and get iovecs back. > Which, as I understand it, you propose? I didn't think of apr_bucket_readv(), more focused on file_bucket_read() to do this internally/transparently, but once file buckets can do that I think we can generalize the concept, at worse filling only iovec[0] when it's ENOTIMPL and/or makes no sense... That'd help the mod_ssl case with something like apr_bucket_readv(min_size=16K), I'll try to think of it once/if I can have a more simple to do file_bucket_read() only working ;)
Re: httpd 2.4.25, mpm_event, ssl: segfaults
> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : > > On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: >> On 02/22/2017 12:00 AM, Stefan Eissing wrote: >>> >>> Just so I do not misunderstand: >>> >>> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you >>> are testing? >> >> >> Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable >> the mmap-optimization by default while still improving overall performance >> for most users. >> >> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just >> defines a new buffer size for use with the file bucket. It's a little less >> than 64K, I assume to make room for an allocation header: >> >>#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) > > Actually I'm not very pleased with this solution (or the final one > that would make this size open / configurable). > The issue is potentially the huge (order big-n) allocations which > finally may hurt the system (fragmentation, OOM...). > > So I'm thinking of another way to achieve the same with the current > APR_BUCKET_BUFF_SIZE (2 pages) per alloc. > > The idea is to have a new apr_allocator_allocv() function which would > fill an iovec with what's available in the allocator's freelist (i.e. > spare apr_memnodes) of at least the given min_size bytes (possibly a > max too but I don't see the need for now) and up to the size of the > given iovec. Interesting. Not only for pure files maybe. It would be great if there'd be a SSL_writev()...but until there is, the TLS case will either turn every iovec into its own TLS record or one needs another copy before that. This last strategy is used by mod_http2. Since there are 9 header bytes per frame, copying data into a right sized buffer gives better performance. So, it would be nice to read n bytes from a bucket brigade and get iovecs back. Which, as I understand it, you propose? > This function could be the base of a new apr_bucket_allocv() (and > possibly apr_p[c]allocv(), though out of scope here) which in turn > could be used by file_bucket_read() to get an iovec of available > buffers. > This iovec could then be passed to (new still) apr_file_readv() based > on the readv() syscall, which would allow to read much more data in > one go. > > With this the scheme we'd have iovec from end to end, well, sort of > since mod_ssl would be break the chain but still produce transient > buckets on output which anyway will end up in the core_output_filter's > brigade of aside heap buckets, for apr_socket_sendv() to finally > writev() them. > > We'd also have more recycled heap buckets (hence memnodes in the > allocator) as the core_output_filter retains buckets, all with > APR_BUCKET_BUFF_SIZE, up to THRESHOLD_MAX_BUFFER which, if > configurable and along with MaxMemFree, would be the real limiter of > recycling. > So it's also adaptative. > > Actually it looks like what we need, but I'd like to have feedbacks > before I go further the prototype I have so far (quite straightforward > apr_allocator changes...). > > Thoughts? > > > Regards, > Yann. Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster www.greenbytes.de
Re: httpd 2.4.25, mpm_event, ssl: segfaults
On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: > On 02/22/2017 12:00 AM, Stefan Eissing wrote: >> >> Just so I do not misunderstand: >> >> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you >> are testing? > > > Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable > the mmap-optimization by default while still improving overall performance > for most users. > > Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just > defines a new buffer size for use with the file bucket. It's a little less > than 64K, I assume to make room for an allocation header: > > #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) Actually I'm not very pleased with this solution (or the final one that would make this size open / configurable). The issue is potentially the huge (order big-n) allocations which finally may hurt the system (fragmentation, OOM...). So I'm thinking of another way to achieve the same with the current APR_BUCKET_BUFF_SIZE (2 pages) per alloc. The idea is to have a new apr_allocator_allocv() function which would fill an iovec with what's available in the allocator's freelist (i.e. spare apr_memnodes) of at least the given min_size bytes (possibly a max too but I don't see the need for now) and up to the size of the given iovec. This function could be the base of a new apr_bucket_allocv() (and possibly apr_p[c]allocv(), though out of scope here) which in turn could be used by file_bucket_read() to get an iovec of available buffers. This iovec could then be passed to (new still) apr_file_readv() based on the readv() syscall, which would allow to read much more data in one go. With this the scheme we'd have iovec from end to end, well, sort of since mod_ssl would be break the chain but still produce transient buckets on output which anyway will end up in the core_output_filter's brigade of aside heap buckets, for apr_socket_sendv() to finally writev() them. We'd also have more recycled heap buckets (hence memnodes in the allocator) as the core_output_filter retains buckets, all with APR_BUCKET_BUFF_SIZE, up to THRESHOLD_MAX_BUFFER which, if configurable and along with MaxMemFree, would be the real limiter of recycling. So it's also adaptative. Actually it looks like what we need, but I'd like to have feedbacks before I go further the prototype I have so far (quite straightforward apr_allocator changes...). Thoughts? Regards, Yann.
Re: Expr's lists evaluated in a string context (was: [users@httpd] mod_lua and subprocess_env)
Would it be possible that people are using it that way for exactly that purpose? I am wondering about this causing some regression for external users. > On Feb 22, 2017, at 10:51 AM, Yann Ylavic wrote: > >> On Tue, Feb 21, 2017 at 6:32 PM, Yann Ylavic wrote: >>> >>> Header set Client-SAN "expr=%{PeerExtList:2.5.29.17}" > > This currently fails because list functions are not recognized in a > string context. > > For now, lists can be either expressed with the syntax "{ , > , ... }", with being itself something powerful, or > obtained from the mod_ssl's PeerExtList() function (grab anything from > a peer certificate). > > For the latter case (or for future functions), it could be useful to > be able to work on such strings (e.g. with extracting regexes). > So I wonder if we could return the string elements separated by > something in the case of lists evaluated in a string context. > > For example, the attached patch uses the seprator ", " (quite HTTP > field inspired), but it could be a json string or whatever... > We could also have an explicit tostring/tojson() function which would > stringify anything as argument. > > Or yet more operators on lists, like list_empty(), list_first(), > list_last(), list_nth(), list_match(, ) (returning > another list of matching entries), ... you name it. > > Working on anything from a certificates looks very useful at least. > > WDYT? >
Re: mod_proxy_http2 sni ?
Sent to your email address. On Thursday 23/02/2017 at 10:24, Stefan Eissing wrote: Steffen, can you get a http2:trace2 log of such a curl request? Am 22.02.2017 um 12:52 schrieb Steffen : Picking up the good host now. With a download chrome says in Status bar: starting, and stays there. Curl hanging: 0 14.4M0 327670 0 27237 0 0:09:17 0:00:01 0:09:16 27237 0 14.4M0 327670 0 14873 0 0:17:01 0:00:02 0:16:59 14873 0 14.4M0 327670 0 10230 0 0:24:44 0:00:03 0:24:41 10230 0 14.4M0 327670 0 7796 0 0:32:28 0:00:04 0:32:24 7796 0 14.4M0 327670 0 6297 0 0:40:12 0:00:05 0:40:07 6297 0 14.4M0 327670 0 5282 0 0:47:55 0:00:06 0:47:49 0 0 14.4M0 327670 0 4539 0 0:55:46 0:00:07 0:55:39 0 0 14.4M0 327670 0 3987 0 1:03:29 0:00:08 1:03:21 0 0 14.4M0 327670 0 3554 0 1:11:13 0:00:09 1:11:04 0 0 14.4M0 327670 0 3206 0 1:18:57 0:00:10 1:18:47 0 0 14.4M0 327670 0 2920 0 1:26:41 0:00:11 1:26:30 0 Looks like that the issue is that the front is h2 and the back h2c. On Wednesday 22/02/2017 at 11:30, Stefan Eissing wrote: You can try now in v1.9.1 if it works as needed now. Am 17.02.2017 um 16:11 schrieb Steffen : Looks like the same, is not looking for the host as with 1.1 It is on my wish list for 2.4.26 Op 16 feb. 2017 om 11:38 heeft Stefan Eissing het volgende geschreven: Is this the same as https://github.com/icing/mod_h2/issues/124 ? It seems that the ProxyPreserveHost is not (correctly) implemented. Am 16.02.2017 um 10:42 schrieb Steffen : Have an Apache ssl only in front of an Apache on port 80 with several vhosts. In front have: ProtocolsHonorOrder On Protocols h2 http/1.1 LoadModule http2_module modules/mod_http2.so ProxyPass / http://127.0.0.1:80/ ProxyPassReverse / http://127.0.0.1:80/ In backend have: ProtocolsHonorOrder On Protocols h2c http/1.1 LoadModule http2_module modules/mod_http2.so This is working great and with all the vhosts. When I add/change the front to: ProtocolsHonorOrder On Protocols h2 http/1.1 LoadModule http2_module modules/mod_http2.so LoadModule proxy_http2_module modules/mod_proxy_http2.so ProxyPass / h2c://127.0.0.1:80/ ProxyPassReverse / h2c://127.0.0.1:80/ This is not working as expected, all is going to the default/first vhost. a log line from the backend gives is all cases not found . default 127.0.0.1 - - [16/Feb/2017:10:22:00 +0100] "GET /index.php HTTP/2.0" 404 207 ... Cheers, Steffenal Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster http://www.greenbytes.de Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster http://www.greenbytes.de Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster http://www.greenbytes.de
Re: mod_proxy_http2 sni ?
Steffen, can you get a http2:trace2 log of such a curl request? > Am 22.02.2017 um 12:52 schrieb Steffen : > > Picking up the good host now. > > With a download chrome says in Status bar: starting, and stays there. > > Curl hanging: > > 0 14.4M0 327670 0 27237 0 0:09:17 0:00:01 0:09:16 27237 > 0 14.4M0 327670 0 14873 0 0:17:01 0:00:02 0:16:59 14873 > 0 14.4M0 327670 0 10230 0 0:24:44 0:00:03 0:24:41 10230 > 0 14.4M0 327670 0 7796 0 0:32:28 0:00:04 0:32:24 7796 > 0 14.4M0 327670 0 6297 0 0:40:12 0:00:05 0:40:07 6297 > 0 14.4M0 327670 0 5282 0 0:47:55 0:00:06 0:47:49 0 > 0 14.4M0 327670 0 4539 0 0:55:46 0:00:07 0:55:39 0 > 0 14.4M0 327670 0 3987 0 1:03:29 0:00:08 1:03:21 0 > > > > > 0 14.4M0 327670 0 3554 0 1:11:13 0:00:09 1:11:04 0 > 0 14.4M0 327670 0 3206 0 1:18:57 0:00:10 1:18:47 0 > 0 14.4M0 327670 0 2920 0 1:26:41 0:00:11 1:26:30 0 > > Looks like that the issue is that the front is h2 and the back h2c. > > On Wednesday 22/02/2017 at 11:30, Stefan Eissing wrote: >> You can try now in v1.9.1 if it works as needed now. >> >>> Am 17.02.2017 um 16:11 schrieb Steffen : >>> >>> Looks like the same, is not looking for the host as with 1.1 >>> >>> It is on my wish list for 2.4.26 >>> >>> >>> Op 16 feb. 2017 om 11:38 heeft Stefan Eissing het volgende geschreven: Is this the same as https://github.com/icing/mod_h2/issues/124 ? It seems that the ProxyPreserveHost is not (correctly) implemented. > Am 16.02.2017 um 10:42 schrieb Steffen : > > > Have an Apache ssl only in front of an Apache on port 80 with several > vhosts. > > In front have: > > > ProtocolsHonorOrder On > Protocols h2 http/1.1 > LoadModule http2_module modules/mod_http2.so > > > > ProxyPass / http://127.0.0.1:80/ > ProxyPassReverse / http://127.0.0.1:80/ > > > In backend have: > > > ProtocolsHonorOrder On > Protocols h2c http/1.1 > LoadModule http2_module modules/mod_http2.so > > This is working great and with all the vhosts. > > > When I add/change the front to: > > > ProtocolsHonorOrder On > Protocols h2 http/1.1 > LoadModule http2_module modules/mod_http2.so > LoadModule proxy_http2_module modules/mod_proxy_http2.so > > > ProxyPass / h2c://127.0.0.1:80/ > ProxyPassReverse / h2c://127.0.0.1:80/ > > > This is not working as expected, all is going to the default/first vhost. > > a log line from the backend gives is all cases not found . > > default 127.0.0.1 - - [16/Feb/2017:10:22:00 +0100] "GET /index.php > HTTP/2.0" 404 207 ... > > > Cheers, > > Steffenal > > Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster http://www.greenbytes.de >>> >> >> Stefan Eissing >> >> bytes GmbH >> Hafenstrasse 16 >> 48155 Münster >> http://www.greenbytes.de >> > > Stefan Eissing bytes GmbH Hafenstrasse 16 48155 Münster www.greenbytes.de