Re: svn commit: r1784228 - in /httpd/httpd/trunk: docs/manual/mod/mod_proxy_hcheck.xml modules/proxy/mod_proxy_hcheck.c

[Yann Ylavic]

On Fri, Feb 24, 2017 at 5:49 AM,   wrote:
> Author: jim
> Date: Fri Feb 24 04:49:38 2017
> New Revision: 1784228
>
> URL: http://svn.apache.org/viewvc?rev=1784228=rev
> Log:
> Instead of thread-pools for each server/context, use a server-wide
> top-level threadpool, which is more efficient.
>
> Modified:
> httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c
>
> Modified: httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c
> URL: 
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c?rev=1784228=1784227=1784228=diff
> ==
> --- httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c (original)
> +++ httpd/httpd/trunk/modules/proxy/mod_proxy_hcheck.c Fri Feb 24 04:49:38 
> 2017
>
> @@ -86,13 +84,15 @@ static void *hc_create_config(apr_pool_t
>  ctx->templates = apr_array_make(p, 10, sizeof(hc_template_t));
>  ctx->conditions = apr_table_make(p, 10);
>  ctx->hcworkers = apr_hash_make(p);
> -ctx->tpsize = HC_THREADPOOL_SIZE;
>  ctx->s = s;
>
>  return ctx;
>  }
>
>  static ap_watchdog_t *watchdog;
> +static apr_thread_pool_t *hctp = NULL;
> +static int tpsize = HC_THREADPOOL_SIZE;

Maybe init 'tpsize' in pre_config for --enable-proxy_check=static restarts?


Regards,
Yann.

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 10:06 PM, Daniel Lescohier
 wrote:
> Why would high-order memory allocations be a problem in userspace code,
> which is using virtual memory?  I thought high-order allocations is a big
> problem in kernel space, which has to deal with physical pages.

Well, both in kernel or user space, the difficulty is finding large
contiguous memory.

With virtual memory (admittedly virtually larger than physical
memory), it needs more "active" regions to fail, but still it can fail
if many heterogeneous chunks are to be mapped at a time, the OOM
killer will do its job.

It depends on how closed to the resident memory limit you are of
course (it won't happen if some memory can be compressed or swapped),
but large chunks are no better with lot of RAM either.

>
> But when you write to a socket, doesn't the kernel scatter the userspace
> buffer into multiple SKBs?  SKBs on order-0 pages allocated by the kernel?

Right, in Linux network stack (or drivers) is mainly using (or is
moving to) order 0 or 1 chunks (with scatterlists when needed).

But this is where httpd's job end, we talk about before this here :)

>From the other message:

On Wed, Feb 22, 2017 at 8:55 PM, Daniel Lescohier wrote:
>
> IOW:
> read():Three copies: copy from filesystem cache to httpd read() buffer to
> encrypted-data buffer to kernel socket buffer.

Not really, "copy from filesystem cache to httpd read() buffer" is
likely mapping to userspace, so no copy (on read) here.

> mmap(): Two copies: filesystem page already mapped into httpd, so just copy
> from filesystem (cached) page to encrypted-data buffer to kernel socket
> buffer.

So, as you said earlier the "write to socket" isn't a copy either,
hence both read() and mmap() implementations could work with a single
copy when mod_ssl is involved (this is more than a copy but you
counted it above so), and no copy at all without it.


Regards,
Yann.

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 8:50 PM, Jacob Champion  wrote:
> On 02/22/2017 02:16 PM, Niklas Edmundsson wrote:
>
> I don't think s_server is particularly optimized for performance anyway.
>
> Oh, and just to complete my local testing table:
>
> - test server, writing from memory: 1.2 GiB/s
> - test server, mmap() from disk: 1.1 GiB/s
> - test server, 64K read()s from disk: 1.0 GiB/s
> - httpd trunk with `EnableMMAP on` and serving from disk: 850 MiB/s
> - httpd trunk with 'EnableMMAP off': 580 MiB/s
> - httpd trunk with my no-mmap-64K-block file bucket: 810 MiB/s
>
> My test server's read() implementation is a really naive "block on read,
> then block on write, repeat" loop, so there's probably some improvement to
> be had there, but this is enough proof in my mind that there are major gains
> to be made regardless.

Does no-mmap-2MB-block beats MMap on?

>
>> Going off on a tangent here:
>>
>> For those of you who actually know how the ssl stuff really works, is it
>> possible to get multiple threads involved in doing the encryption, or do
>> you need the results from the previous block in order to do the next
>> one?
>
> I'm not a cryptographer, but I think how parallelizable it is depends on the
> ciphersuite in use. Like you say, some ciphersuites require one block to be
> fed into the next as an input; others don't.

Yes, and the cost of scheduling threads for non dedicated cypto device
is not worth it I think.
But mainly there is not only one stream involved in a typical HTTP
server, so probably multiple simulteneous connections already saturate
the AES-NI...

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 7:16 PM, Jacob Champion  wrote:
> On 02/23/2017 08:34 AM, Yann Ylavic wrote:
>> Actually I'm not very pleased with this solution (or the final one
>> that would make this size open / configurable).
>> The issue is potentially the huge (order big-n) allocations which
>> finally may hurt the system (fragmentation, OOM...).
>
> Power users can break the system, and this is a power tool, right?

It's not about power users, I don't think we can recommend anyone to
use 4MB buffers even if they (seem to) have RAM for it.

> And we
> have HugeTLB kernels and filesystems to play with, with 2MB and bigger
> pages... Making all these assumptions for 90% of users is perfect for the
> out-of-the-box experience, but hardcoding them so that no one can fix broken
> assumptions seems Bad.

I'm not talking about hardcoding anything, nor reading or sendind hard
limited sizes on filesystem/sockets.
I'm proposing that the configuration relates to how much we "coalesce"
data on output, and all buffers' reuses (though each of fixed size)
should follow the needs.

>
> (And don't get me wrong, I think applying vectored I/O to the brigade would
> be a great thing to try out and benchmark. I just think it's a long-term and
> heavily architectural fix, when a short-term change to get rid of some
> #defined constants could have immediate benefits.)

Of course, apr_bucket_file_set_read_size() is a quick patch (I dispute
it for the general case, but it may be useful, or not, for the 16K SSL
case, let's see), and so is another for configuring core_output_filter
constants, but we don't need them for testing right?

>
>> I've no idea how much it costs to have 8K vs 16K records, though.
>> Maybe in the mod_ssl case we'd want 16K buffers, still reasonable?
>
>
> We can't/shouldn't hardcode this especially. People who want maximum
> throughput may want nice big records, but IIRC users who want progressive
> rendering need smaller records so that they don't have to wait as long for
> the first decrypted chunk. It needs to be tunable, possibly per-location.

Well, the limit is 16K at the TLS level.


Regards,
Yann.

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 7:15 PM, Niklas Edmundsson  wrote:
> On Thu, 23 Feb 2017, Yann Ylavic wrote:
>
>>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
>>> defines a new buffer size for use with the file bucket. It's a little
>>> less
>>> than 64K, I assume to make room for an allocation header:
>>>
>>> #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)
>>
>>
>> Actually I'm not very pleased with this solution (or the final one
>> that would make this size open / configurable).
>> The issue is potentially the huge (order big-n) allocations which
>> finally may hurt the system (fragmentation, OOM...).
>
>
> Is this a real or theoretical problem?

Both. Fragmentation is a hard issue, but a constant is: the more you
ask for big allocs, the more likely you won't be serviced one day or
another (or another task will be killed for that, until yours).

Long living (or closed to memory limits) systems suffer from this, no
matter what allocator, small and large allocations fragment the memory
(httpd is likely not the only program on the system), the only
"remedy" is small order allocations (2^order pages, a "sane" order
being lower than 2, hence order-1 on a system with 4K pages is 8K
bytes).

>
> Our large-file cache module does 128k allocs to get a sane block size when
> copying files to the cache. The only potential drawback we noticed was httpd
> processes becoming bloated due to the default MaxMemFree 2048, so we're
> running with MaxMemFree 256 now.

With MaxMemFree 256 (KB per allocator), each APR allocator reclaims
but mainly returns a lot more memory chunks to the system's allocator,
which does a better job to in recycling many differently sized chunks
than APR's (which is pretty basic in this regard, it's role is more
about quickly recycling common sizes).

With MaxMemFree 2048 (2MB), there is more builtin handling of "exotic"
chunks, which may be painful.

That might be something else, though...

>
> Granted, doing alloc/free for all outgoing data means way more alloc/free:s,
> so we might just miss the issues because cache fills aren't as common.

That's why reuse of common and reasonably sized chunks in the APR
allocator can help.

>
> However, for large file performance I really don't buy into the notion that
> it's a good idea to break everything into tiny puny blocks. The potential
> for wasting CPU cycles on this micro-management is rather big...

I don't think that a readv/writev of 16 iovecs of 8KB is (noticeably)
slower than read/write of contiguous 128K, both might well end up in a
scaterlist for the kernel/hardware.

>
> I do find iovecs useful, it the small blocks that gets me into skeptic
> mode...

Small blocks is not for networking, it's for internal use only.
And remember that TLS records are 16K max anyway, give 128KB to
openssl's SSL_write it will output 8 chunks of 16KB.

>
>
> Kinda related: We also have the support for larger page sizes with modern
> CPUs. Has anyone investigated if it makes sense allocating memory pools in
> chunks that fit those large pages?

I think PPC64 have 64K pages already.
APR pools are already based on the page size IIRC.


Regards,
Yann.

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Daniel Lescohier]

Why would high-order memory allocations be a problem in userspace code,
which is using virtual memory?  I thought high-order allocations is a big
problem in kernel space, which has to deal with physical pages.

But when you write to a socket, doesn't the kernel scatter the userspace
buffer into multiple SKBs?  SKBs on order-0 pages allocated by the kernel?


On Thu, Feb 23, 2017 at 1:16 PM, Jacob Champion 
wrote:

> On 02/23/2017 08:34 AM, Yann Ylavic wrote:
> > Actually I'm not very pleased with this solution (or the final one
> > that would make this size open / configurable).
> > The issue is potentially the huge (order big-n) allocations which
> > finally may hurt the system (fragmentation, OOM...).
>
> Power users can break the system, and this is a power tool, right? And we
> have HugeTLB kernels and filesystems to play with, with 2MB and bigger
> pages... Making all these assumptions for 90% of users is perfect for the
> out-of-the-box experience, but hardcoding them so that no one can fix
> broken assumptions seems Bad.
>
> (And don't get me wrong, I think applying vectored I/O to the brigade
> would be a great thing to try out and benchmark. I just think it's a
> long-term and heavily architectural fix, when a short-term change to get
> rid of some #defined constants could have immediate benefits.)
>
> I've no idea how much it costs to have 8K vs 16K records, though.
>> Maybe in the mod_ssl case we'd want 16K buffers, still reasonable?
>>
>
> We can't/shouldn't hardcode this especially. People who want maximum
> throughput may want nice big records, but IIRC users who want progressive
> rendering need smaller records so that they don't have to wait as long for
> the first decrypted chunk. It needs to be tunable, possibly per-location.
>
> --Jacob
>

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Jacob Champion]

On 02/22/2017 02:16 PM, Niklas Edmundsson wrote:

Any joy with something simpler like gprof? (Caveat: haven't used it in
ages to I don't know if its even applicable nowadays).


Well, if I had thought about it a little more, I would have remembered 
that instrumenting profilers don't profile syscalls very well, and they 
especially mess with I/O timing. Valgrind was completely inconclusive on 
the read() vs. mmap() front. :(


(...except that it showed that a good 25% of my test server's CPU time 
was spent inside OpenSSL in a memcpy(). Interesting...)



So httpd isn't beat by the naive openssl s_server approach at least ;-)


I don't think s_server is particularly optimized for performance anyway.

Oh, and just to complete my local testing table:

- test server, writing from memory: 1.2 GiB/s
- test server, mmap() from disk: 1.1 GiB/s
- test server, 64K read()s from disk: 1.0 GiB/s
- httpd trunk with `EnableMMAP on` and serving from disk: 850 MiB/s
- httpd trunk with 'EnableMMAP off': 580 MiB/s
- httpd trunk with my no-mmap-64K-block file bucket: 810 MiB/s

My test server's read() implementation is a really naive "block on read, 
then block on write, repeat" loop, so there's probably some improvement 
to be had there, but this is enough proof in my mind that there are 
major gains to be made regardless.



Going off on a tangent here:

For those of you who actually know how the ssl stuff really works, is it
possible to get multiple threads involved in doing the encryption, or do
you need the results from the previous block in order to do the next
one?


I'm not a cryptographer, but I think how parallelizable it is depends on 
the ciphersuite in use. Like you say, some ciphersuites require one 
block to be fed into the next as an input; others don't.


--Jacob

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Jacob Champion]

On 02/23/2017 08:34 AM, Yann Ylavic wrote:
> Actually I'm not very pleased with this solution (or the final one
> that would make this size open / configurable).
> The issue is potentially the huge (order big-n) allocations which
> finally may hurt the system (fragmentation, OOM...).

Power users can break the system, and this is a power tool, right? And 
we have HugeTLB kernels and filesystems to play with, with 2MB and 
bigger pages... Making all these assumptions for 90% of users is perfect 
for the out-of-the-box experience, but hardcoding them so that no one 
can fix broken assumptions seems Bad.


(And don't get me wrong, I think applying vectored I/O to the brigade 
would be a great thing to try out and benchmark. I just think it's a 
long-term and heavily architectural fix, when a short-term change to get 
rid of some #defined constants could have immediate benefits.)



I've no idea how much it costs to have 8K vs 16K records, though.
Maybe in the mod_ssl case we'd want 16K buffers, still reasonable?


We can't/shouldn't hardcode this especially. People who want maximum 
throughput may want nice big records, but IIRC users who want 
progressive rendering need smaller records so that they don't have to 
wait as long for the first decrypted chunk. It needs to be tunable, 
possibly per-location.


--Jacob

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Niklas Edmundsson]

On Thu, 23 Feb 2017, Yann Ylavic wrote:


Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
defines a new buffer size for use with the file bucket. It's a little less
than 64K, I assume to make room for an allocation header:

#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)


Actually I'm not very pleased with this solution (or the final one
that would make this size open / configurable).
The issue is potentially the huge (order big-n) allocations which
finally may hurt the system (fragmentation, OOM...).


Is this a real or theoretical problem?

Our large-file cache module does 128k allocs to get a sane block size 
when copying files to the cache. The only potential drawback we 
noticed was httpd processes becoming bloated due to the default 
MaxMemFree 2048, so we're running with MaxMemFree 256 now. I don't 
know if things got much better, but it isn't breaking anything 
either...


Granted, doing alloc/free for all outgoing data means way more 
alloc/free:s, so we might just miss the issues because cache fills 
aren't as common.


However, for large file performance I really don't buy into the notion 
that it's a good idea to break everything into tiny puny blocks. The 
potential for wasting CPU cycles on this micro-management is rather 
big...


I can see it working for a small-file workload where files aren't much 
bigger than tens of kB anyway, but not so much for large-file 
delivery.


A prudent way forward might be to investigate what impact different 
block sizes have wrt ssl/https first.


As networking speeds go up it is kind of expected that block sizes 
needs to go up as well, especially as per-core clock frequency isn't 
increasing much (it's been at 2-ish GHz base frequency for server CPUs 
the last ten years now?) and we're relying more and more on various 
offload mechanisms in CPUs/NICs etc to get us from 1 Gbps to 10 Gbps 
to 100 Gbps ...


I do find iovecs useful, it the small blocks that gets me into skeptic 
mode...



Kinda related: We also have the support for larger page sizes with 
modern CPUs. Has anyone investigated if it makes sense allocating 
memory pools in chunks that fit those large pages?




/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
 Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se  | ni...@acc.umu.se
---
 You need not worry about your future
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 5:34 PM, Yann Ylavic  wrote:
> On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing
>  wrote:
>>
>>> Am 23.02.2017 um 16:38 schrieb Yann Ylavic :
>>>
>>> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion  
>>> wrote:
 On 02/22/2017 12:00 AM, Stefan Eissing wrote:
>
> Just so I do not misunderstand:
>
> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you
> are testing?


 Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable
 the mmap-optimization by default while still improving overall performance
 for most users.

 Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
 defines a new buffer size for use with the file bucket. It's a little less
 than 64K, I assume to make room for an allocation header:

#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)
>>>
>>> Actually I'm not very pleased with this solution (or the final one
>>> that would make this size open / configurable).
>>> The issue is potentially the huge (order big-n) allocations which
>>> finally may hurt the system (fragmentation, OOM...).
>>>
>>> So I'm thinking of another way to achieve the same with the current
>>> APR_BUCKET_BUFF_SIZE (2 pages) per alloc.
>>>
>>> The idea is to have a new apr_allocator_allocv() function which would
>>> fill an iovec with what's available in the allocator's freelist (i.e.
>>> spare apr_memnodes) of at least the given min_size bytes (possibly a
>>> max too but I don't see the need for now) and up to the size of the
>>> given iovec.
>>
>> Interesting. Not only for pure files maybe.
>>
>> It would be great if there'd be a SSL_writev()...
>
> Indeed, openssl would fully fill the TLS records.
>
>> but until there is, the TLS case will either turn every iovec into
>> its own TLS record
>
> Yes, but it's more a client issue to work with these records finally
> (because from there all is networking only, and coalescing will happen
> in the core output filter from a socket POV).
>
> Anyway mod_proxy(s) could be the client, so...
>
> I've no idea how much it costs to have 8K vs 16K records, though.
> Maybe in the mod_ssl case we'd want 16K buffers, still reasonable?
>
>> or one needs another copy before that. This last strategy is used by
>> mod_http2. Since there are 9 header bytes per frame, copying data
>> into a right sized buffer gives better performance. So, it would be
>> nice to read n bytes from a bucket brigade and get iovecs back.
>> Which, as I understand it, you propose?
>
> I didn't think of apr_bucket_readv(), more focused on
> file_bucket_read() to do this internally/transparently, but once file
> buckets can do that I think we can generalize the concept, at worse
> filling only iovec[0] when it's ENOTIMPL and/or makes no sense...
>
> That'd help the mod_ssl case with something like
> apr_bucket_readv(min_size=16K), I'll try to think of it once/if I can
> have a more simple to do file_bucket_read() only working ;)

Hm no, this needs to happen on the producer side, not at final
apr_bucket_read().
So for the mod_ssl case I think we could have a simple (new)
apr_bucket_alloc_set_size(16K) in the first place if it helps more
than hurts.

As for your question about "iovecs back" (which I finally didn't
answer), all happens at the bucket alloc level, when buckets are
deleted.

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing
 wrote:
>
>> Am 23.02.2017 um 16:38 schrieb Yann Ylavic :
>>
>> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion  wrote:
>>> On 02/22/2017 12:00 AM, Stefan Eissing wrote:

 Just so I do not misunderstand:

 you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you
 are testing?
>>>
>>>
>>> Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable
>>> the mmap-optimization by default while still improving overall performance
>>> for most users.
>>>
>>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
>>> defines a new buffer size for use with the file bucket. It's a little less
>>> than 64K, I assume to make room for an allocation header:
>>>
>>>#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)
>>
>> Actually I'm not very pleased with this solution (or the final one
>> that would make this size open / configurable).
>> The issue is potentially the huge (order big-n) allocations which
>> finally may hurt the system (fragmentation, OOM...).
>>
>> So I'm thinking of another way to achieve the same with the current
>> APR_BUCKET_BUFF_SIZE (2 pages) per alloc.
>>
>> The idea is to have a new apr_allocator_allocv() function which would
>> fill an iovec with what's available in the allocator's freelist (i.e.
>> spare apr_memnodes) of at least the given min_size bytes (possibly a
>> max too but I don't see the need for now) and up to the size of the
>> given iovec.
>
> Interesting. Not only for pure files maybe.
>
> It would be great if there'd be a SSL_writev()...

Indeed, openssl would fully fill the TLS records.

> but until there is, the TLS case will either turn every iovec into
> its own TLS record

Yes, but it's more a client issue to work with these records finally
(because from there all is networking only, and coalescing will happen
in the core output filter from a socket POV).

Anyway mod_proxy(s) could be the client, so...

I've no idea how much it costs to have 8K vs 16K records, though.
Maybe in the mod_ssl case we'd want 16K buffers, still reasonable?

> or one needs another copy before that. This last strategy is used by
> mod_http2. Since there are 9 header bytes per frame, copying data
> into a right sized buffer gives better performance. So, it would be
> nice to read n bytes from a bucket brigade and get iovecs back.
> Which, as I understand it, you propose?

I didn't think of apr_bucket_readv(), more focused on
file_bucket_read() to do this internally/transparently, but once file
buckets can do that I think we can generalize the concept, at worse
filling only iovec[0] when it's ENOTIMPL and/or makes no sense...

That'd help the mod_ssl case with something like
apr_bucket_readv(min_size=16K), I'll try to think of it once/if I can
have a more simple to do file_bucket_read() only working ;)

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Stefan Eissing]

> Am 23.02.2017 um 16:38 schrieb Yann Ylavic :
> 
> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion  wrote:
>> On 02/22/2017 12:00 AM, Stefan Eissing wrote:
>>> 
>>> Just so I do not misunderstand:
>>> 
>>> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you
>>> are testing?
>> 
>> 
>> Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable
>> the mmap-optimization by default while still improving overall performance
>> for most users.
>> 
>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
>> defines a new buffer size for use with the file bucket. It's a little less
>> than 64K, I assume to make room for an allocation header:
>> 
>>#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)
> 
> Actually I'm not very pleased with this solution (or the final one
> that would make this size open / configurable).
> The issue is potentially the huge (order big-n) allocations which
> finally may hurt the system (fragmentation, OOM...).
> 
> So I'm thinking of another way to achieve the same with the current
> APR_BUCKET_BUFF_SIZE (2 pages) per alloc.
> 
> The idea is to have a new apr_allocator_allocv() function which would
> fill an iovec with what's available in the allocator's freelist (i.e.
> spare apr_memnodes) of at least the given min_size bytes (possibly a
> max too but I don't see the need for now) and up to the size of the
> given iovec.

Interesting. Not only for pure files maybe.

It would be great if there'd be a SSL_writev()...but until there is, the TLS 
case will either turn every iovec into its own TLS record or one needs another 
copy before that. This last strategy is used by mod_http2. Since there are 9 
header bytes per frame, copying data into a right sized buffer gives better 
performance. So, it would be nice to read n bytes from a bucket brigade and get 
iovecs back. Which, as I understand it, you propose?

> This function could be the base of a new apr_bucket_allocv() (and
> possibly apr_p[c]allocv(), though out of scope here) which in turn
> could be used by file_bucket_read() to get an iovec of available
> buffers.
> This iovec could then be passed to (new still) apr_file_readv() based
> on the readv() syscall, which would allow to read much more data in
> one go.
> 
> With this the scheme we'd have iovec from end to end, well, sort of
> since mod_ssl would be break the chain but still produce transient
> buckets on output which anyway will end up in the core_output_filter's
> brigade of aside heap buckets, for apr_socket_sendv() to finally
> writev() them.
> 
> We'd also have more recycled heap buckets (hence memnodes in the
> allocator) as the core_output_filter retains buckets, all with
> APR_BUCKET_BUFF_SIZE, up to THRESHOLD_MAX_BUFFER which, if
> configurable and along with MaxMemFree, would be the real limiter of
> recycling.
> So it's also adaptative.
> 
> Actually it looks like what we need, but I'd like to have feedbacks
> before I go further the prototype I have so far (quite straightforward
> apr_allocator changes...).
> 
> Thoughts?
> 
> 
> Regards,
> Yann.

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[Yann Ylavic]

On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion  wrote:
> On 02/22/2017 12:00 AM, Stefan Eissing wrote:
>>
>> Just so I do not misunderstand:
>>
>> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you
>> are testing?
>
>
> Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable
> the mmap-optimization by default while still improving overall performance
> for most users.
>
> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just
> defines a new buffer size for use with the file bucket. It's a little less
> than 64K, I assume to make room for an allocation header:
>
> #define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64)

Actually I'm not very pleased with this solution (or the final one
that would make this size open / configurable).
The issue is potentially the huge (order big-n) allocations which
finally may hurt the system (fragmentation, OOM...).

So I'm thinking of another way to achieve the same with the current
APR_BUCKET_BUFF_SIZE (2 pages) per alloc.

The idea is to have a new apr_allocator_allocv() function which would
fill an iovec with what's available in the allocator's freelist (i.e.
spare apr_memnodes) of at least the given min_size bytes (possibly a
max too but I don't see the need for now) and up to the size of the
given iovec.

This function could be the base of a new apr_bucket_allocv() (and
possibly apr_p[c]allocv(), though out of scope here) which in turn
could be used by file_bucket_read() to get an iovec of available
buffers.
This iovec could then be passed to (new still) apr_file_readv() based
on the readv() syscall, which would allow to read much more data in
one go.

With this the scheme we'd have iovec from end to end, well, sort of
since mod_ssl would be break the chain but still produce transient
buckets on output which anyway will end up in the core_output_filter's
brigade of aside heap buckets, for apr_socket_sendv() to finally
writev() them.

We'd also have more recycled heap buckets (hence memnodes in the
allocator) as the core_output_filter retains buckets, all with
APR_BUCKET_BUFF_SIZE, up to THRESHOLD_MAX_BUFFER which, if
configurable and along with MaxMemFree, would be the real limiter of
recycling.
So it's also adaptative.

Actually it looks like what we need, but I'd like to have feedbacks
before I go further the prototype I have so far (quite straightforward
apr_allocator changes...).

Thoughts?


Regards,
Yann.

Re: Expr's lists evaluated in a string context (was: [users@httpd] mod_lua and subprocess_env)

[Jim Jagielski]

Would it be possible that people are using it that way
for exactly that purpose? I am wondering about this
causing some regression for external users.

> On Feb 22, 2017, at 10:51 AM, Yann Ylavic  wrote:
> 
>> On Tue, Feb 21, 2017 at 6:32 PM, Yann Ylavic  wrote:
>>> 
>>> Header set Client-SAN "expr=%{PeerExtList:2.5.29.17}"
> 
> This currently fails because list functions are not recognized in a
> string context.
> 
> For now, lists can be either expressed with the syntax "{ ,
> , ... }", with  being itself something powerful, or
> obtained from the mod_ssl's PeerExtList() function (grab anything from
> a peer certificate).
> 
> For the latter case (or for future functions), it could be useful to
> be able to work on such strings (e.g. with extracting regexes).
> So I wonder if we could return the string elements separated by
> something in the case of lists evaluated in a string context.
> 
> For example, the attached patch uses the seprator ", " (quite HTTP
> field inspired), but it could be a json string or whatever...
> We could also have an explicit tostring/tojson() function which would
> stringify anything as argument.
> 
> Or yet more operators on lists, like list_empty(), list_first(),
> list_last(), list_nth(), list_match(, ) (returning
> another list of matching entries), ... you name it.
> 
> Working on anything from a certificates looks very useful at least.
> 
> WDYT?
> 

Re: mod_proxy_http2 sni ?

[Steffen]

Sent to your email address.


On Thursday 23/02/2017 at 10:24, Stefan Eissing  wrote:

Steffen,

can you get a http2:trace2 log of such a curl request?



Am 22.02.2017 um 12:52 schrieb Steffen :

Picking up the good host now.

With a download chrome says in Status bar: starting, and stays there.

Curl hanging:

   0 14.4M0 327670 0  27237  0  0:09:17  0:00:01  
0:09:16 27237
 0 14.4M0 327670 0  14873  0  0:17:01  0:00:02  
0:16:59 14873
 0 14.4M0 327670 0  10230  0  0:24:44  0:00:03  
0:24:41 10230
 0 14.4M0 327670 0   7796  0  0:32:28  0:00:04  
0:32:24  7796
 0 14.4M0 327670 0   6297  0  0:40:12  0:00:05  
0:40:07  6297
 0 14.4M0 327670 0   5282  0  0:47:55  0:00:06  
0:47:49 0
 0 14.4M0 327670 0   4539  0  0:55:46  0:00:07  
0:55:39 0
 0 14.4M0 327670 0   3987  0  1:03:29  0:00:08  
1:03:21 0





 0 14.4M0 327670 0   3554  0  1:11:13  0:00:09  
1:11:04 0
 0 14.4M0 327670 0   3206  0  1:18:57  0:00:10  
1:18:47 0
 0 14.4M0 327670 0   2920  0  1:26:41  0:00:11  
1:26:30 0


Looks like that the issue is that the front is h2 and the back h2c.

On Wednesday 22/02/2017 at 11:30, Stefan Eissing wrote:


You can try now in v1.9.1 if it works as needed now.



Am 17.02.2017 um 16:11 schrieb Steffen :

Looks like the same, is not looking for the host as with 1.1

It is on my wish list for 2.4.26





Op 16 feb. 2017 om 11:38 heeft Stefan Eissing 
 het volgende geschreven:


Is this the same as https://github.com/icing/mod_h2/issues/124 ?

It seems that the ProxyPreserveHost is not (correctly) implemented.



Am 16.02.2017 um 10:42 schrieb Steffen :


Have an Apache ssl only in front of an Apache on port 80 with several 
vhosts.


In front have:


ProtocolsHonorOrder On
Protocols h2 http/1.1
LoadModule http2_module modules/mod_http2.so



ProxyPass / http://127.0.0.1:80/
ProxyPassReverse / http://127.0.0.1:80/


In backend have:


ProtocolsHonorOrder On
Protocols h2c http/1.1
LoadModule http2_module modules/mod_http2.so

This is working great and with all the vhosts.


When I add/change the front to:


ProtocolsHonorOrder On
Protocols h2 http/1.1
LoadModule http2_module modules/mod_http2.so
LoadModule proxy_http2_module modules/mod_proxy_http2.so


ProxyPass / h2c://127.0.0.1:80/
ProxyPassReverse / h2c://127.0.0.1:80/


This is not working as expected, all is going to the default/first 
vhost.


a log line from the backend gives is all cases not found .

default 127.0.0.1 - - [16/Feb/2017:10:22:00 +0100] "GET /index.php 
HTTP/2.0" 404 207 ...



Cheers,

Steffenal




Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
http://www.greenbytes.de





Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
http://www.greenbytes.de






Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
http://www.greenbytes.de

Re: mod_proxy_http2 sni ?

[Stefan Eissing]

Steffen,

can you get a http2:trace2 log of such a curl request?

> Am 22.02.2017 um 12:52 schrieb Steffen :
> 
> Picking up the good host now.
> 
> With a download chrome says in Status bar: starting, and stays there.
> 
> Curl hanging:
> 
>  0 14.4M0 327670 0  27237  0  0:09:17  0:00:01  0:09:16 27237
>   0 14.4M0 327670 0  14873  0  0:17:01  0:00:02  0:16:59 14873
>   0 14.4M0 327670 0  10230  0  0:24:44  0:00:03  0:24:41 10230
>   0 14.4M0 327670 0   7796  0  0:32:28  0:00:04  0:32:24  7796
>   0 14.4M0 327670 0   6297  0  0:40:12  0:00:05  0:40:07  6297
>   0 14.4M0 327670 0   5282  0  0:47:55  0:00:06  0:47:49 0
>   0 14.4M0 327670 0   4539  0  0:55:46  0:00:07  0:55:39 0
>   0 14.4M0 327670 0   3987  0  1:03:29  0:00:08  1:03:21 0
> 
> 
> 
> 
>   0 14.4M0 327670 0   3554  0  1:11:13  0:00:09  1:11:04 0
>   0 14.4M0 327670 0   3206  0  1:18:57  0:00:10  1:18:47 0
>   0 14.4M0 327670 0   2920  0  1:26:41  0:00:11  1:26:30 0
> 
> Looks like that the issue is that the front is h2 and the back h2c.
>  
> On Wednesday 22/02/2017 at 11:30, Stefan Eissing wrote: 
>> You can try now in v1.9.1 if it works as needed now.
>> 
>>> Am 17.02.2017 um 16:11 schrieb Steffen :
>>> 
>>> Looks like the same, is not looking for the host as with 1.1
>>> 
>>> It is on my wish list for 2.4.26 
>>> 
>>> 
>>> 
 Op 16 feb. 2017 om 11:38 heeft Stefan Eissing 
  het volgende geschreven:
 
 Is this the same as https://github.com/icing/mod_h2/issues/124 ?
 
 It seems that the ProxyPreserveHost is not (correctly) implemented.
 
> Am 16.02.2017 um 10:42 schrieb Steffen :
> 
> 
> Have an Apache ssl only in front of an Apache on port 80 with several 
> vhosts.
> 
> In front have:
> 
> 
> ProtocolsHonorOrder On
> Protocols h2 http/1.1
> LoadModule http2_module modules/mod_http2.so
> 
> 
> 
> ProxyPass / http://127.0.0.1:80/
> ProxyPassReverse / http://127.0.0.1:80/
> 
> 
> In backend have:
> 
> 
> ProtocolsHonorOrder On
> Protocols h2c http/1.1
> LoadModule http2_module modules/mod_http2.so
> 
> This is working great and with all the vhosts.
> 
> 
> When I add/change the front to:
> 
> 
> ProtocolsHonorOrder On
> Protocols h2 http/1.1
> LoadModule http2_module modules/mod_http2.so
> LoadModule proxy_http2_module modules/mod_proxy_http2.so
> 
> 
> ProxyPass / h2c://127.0.0.1:80/
> ProxyPassReverse / h2c://127.0.0.1:80/
> 
> 
> This is not working as expected, all is going to the default/first vhost.
> 
> a log line from the backend gives is all cases not found .
> 
> default 127.0.0.1 - - [16/Feb/2017:10:22:00 +0100] "GET /index.php 
> HTTP/2.0" 404 207 ...
> 
> 
> Cheers,
> 
> Steffenal
> 
> 
 
 Stefan Eissing
 
 bytes GmbH
 Hafenstrasse 16
 48155 Münster
 http://www.greenbytes.de
 
>>> 
>> 
>> Stefan Eissing
>> 
>> bytes GmbH
>> Hafenstrasse 16
>> 48155 Münster
>> http://www.greenbytes.de
>> 
> 
> 

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de