Re: Read scattered to gather send (was [too long]: httpd 2.4.25, mpm_event, ssl: segfaults)

This really belongs in dev@apr... Adding it now. If we are *really* drastically changing this core aspect of pool allocation, we should reconsider my thoughts related to more granular mutex related to anytime we alloc from a pool to allow for, if required/desired, full thread safety not just when

Re: Read scattered to gather send (was [too long]: httpd 2.4.25, mpm_event, ssl: segfaults)

Hello Yann, Just one question at this time: It appears that your changes to httpd will depend on new functions in APR-util which would require a new release of APR-util (and APR as well?). What is your plan for handling this dependency? Thanks, Mike Rumph On 3/3/2017 11:38 AM, Yann Ylavic

Re: Read scattered to gather send (was [too long]: httpd 2.4.25, mpm_event, ssl: segfaults)

First fix :) On Fri, Mar 3, 2017 at 6:41 PM, Yann Ylavic wrote: > > With apr_allocator_bulk_alloc(), one can request several apr_memnode_t > of a fixed (optional) or minimal given size, and in the worst case get > a single one (allocaœted), or in the best case as much free

Re: Read scattered to gather send (was [too long]: httpd 2.4.25, mpm_event, ssl: segfaults)

On Fri, Mar 3, 2017 at 6:41 PM, Yann Ylavic wrote: > > Currently I have good results (with gdb/LOG_TRACE, no stress test yet ;) > > For "http:" (main server) with: > > EnableMMAP off > EnableSendfile off > > EnableScatterReadfile on > #FileReadBufferSize 8192

Read scattered to gather send (was [too long]: httpd 2.4.25, mpm_event, ssl: segfaults)

Hi, few free time lately, yet I tried to implement this and it seems to work pretty well. Details below (and 2.4.x patches attached)... On Thu, Feb 23, 2017 at 4:38 PM, Yann Ylavic wrote: > > So I'm thinking of another way to achieve the same with the current >

Re: httpd 2.4.25, mpm_event, ssl: segfaults

All writes to Linux sockets means the kernel copies to 2kiB buffers used by SKBs. It's copied to somewhere in the middle of that 2kiB buffer, so that TCP/IP headers can be prepended by the kernel. Even with TCP Segmentation Offload, 2kiB buffers are still used; it just means that the TCP/IP

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Mon, Feb 27, 2017 at 12:16 PM, Jacob Champion wrote: > > On 02/23/2017 04:48 PM, Yann Ylavic wrote: >> On Wed, Feb 22, 2017 at 8:55 PM, Daniel Lescohier wrote: >>> >>> >>> IOW: read():Three copies: copy from filesystem cache to httpd >>> read() buffer to encrypted-data

Re: httpd 2.4.25, mpm_event, ssl: segfaults

[combining two replies] On 02/23/2017 04:47 PM, Yann Ylavic wrote: On Thu, Feb 23, 2017 at 7:16 PM, Jacob Champion wrote: Power users can break the system, and this is a power tool, right? It's not about power users, I don't think we can recommend anyone to use 4MB

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Fri, 24 Feb 2017, Yann Ylavic wrote: On Thu, Feb 23, 2017 at 8:50 PM, Jacob Champion wrote: Going off on a tangent here: For those of you who actually know how the ssl stuff really works, is it possible to get multiple threads involved in doing the encryption, or do

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Fri, 24 Feb 2017, Yann Ylavic wrote: The issue is potentially the huge (order big-n) allocations which finally may hurt the system (fragmentation, OOM...). Is this a real or theoretical problem? Both. Fragmentation is a hard issue, but a constant is: the more you ask for big allocs, the

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 7:48 PM, Yann Ylavic wrote: > On Wed, Feb 22, 2017 at 8:55 PM, Daniel Lescohier wrote: > > > > IOW: > > read():Three copies: copy from filesystem cache to httpd read() buffer to > > encrypted-data buffer to kernel socket buffer. > > Not really, "copy

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 10:06 PM, Daniel Lescohier wrote: > Why would high-order memory allocations be a problem in userspace code, > which is using virtual memory? I thought high-order allocations is a big > problem in kernel space, which has to deal with physical

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 8:50 PM, Jacob Champion wrote: > On 02/22/2017 02:16 PM, Niklas Edmundsson wrote: > > I don't think s_server is particularly optimized for performance anyway. > > Oh, and just to complete my local testing table: > > - test server, writing from memory:

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 7:16 PM, Jacob Champion wrote: > On 02/23/2017 08:34 AM, Yann Ylavic wrote: >> Actually I'm not very pleased with this solution (or the final one >> that would make this size open / configurable). >> The issue is potentially the huge (order big-n)

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 7:15 PM, Niklas Edmundsson wrote: > On Thu, 23 Feb 2017, Yann Ylavic wrote: > >>> Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just >>> defines a new buffer size for use with the file bucket. It's a little >>> less >>> than 64K, I

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Why would high-order memory allocations be a problem in userspace code, which is using virtual memory? I thought high-order allocations is a big problem in kernel space, which has to deal with physical pages. But when you write to a socket, doesn't the kernel scatter the userspace buffer into

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/22/2017 02:16 PM, Niklas Edmundsson wrote: Any joy with something simpler like gprof? (Caveat: haven't used it in ages to I don't know if its even applicable nowadays). Well, if I had thought about it a little more, I would have remembered that instrumenting profilers don't profile

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/23/2017 08:34 AM, Yann Ylavic wrote: > Actually I'm not very pleased with this solution (or the final one > that would make this size open / configurable). > The issue is potentially the huge (order big-n) allocations which > finally may hurt the system (fragmentation, OOM...). Power users

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 23 Feb 2017, Yann Ylavic wrote: Technically, Yann's patch doesn't redefine APR_BUCKET_BUFF_SIZE, it just defines a new buffer size for use with the file bucket. It's a little less than 64K, I assume to make room for an allocation header: #define FILE_BUCKET_BUFF_SIZE (64 * 1024 -

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 5:34 PM, Yann Ylavic wrote: > On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing > wrote: >> >>> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : >>> >>> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 23, 2017 at 4:58 PM, Stefan Eissing wrote: > >> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : >> >> On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: >>> On 02/22/2017 12:00 AM, Stefan Eissing wrote:

Re: httpd 2.4.25, mpm_event, ssl: segfaults

> Am 23.02.2017 um 16:38 schrieb Yann Ylavic : > > On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: >> On 02/22/2017 12:00 AM, Stefan Eissing wrote: >>> >>> Just so I do not misunderstand: >>> >>> you increased BUCKET_BUFF_SIZE in APR from

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, Feb 22, 2017 at 6:36 PM, Jacob Champion wrote: > On 02/22/2017 12:00 AM, Stefan Eissing wrote: >> >> Just so I do not misunderstand: >> >> you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you >> are testing? > > > Essentially, yes, *and* turn off

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, 22 Feb 2017, Jacob Champion wrote: To make results less confusing, any specific patches/branch I should test? My baseline is httpd-2.4.25 + httpd-2.4.25-deps --with-included-apr FWIW. 2.4.25 is just fine. We'll have to make sure there's nothing substantially different about it

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, Feb 22, 2017 at 2:42 PM, Jacob Champion wrote: > Ah, but they *do*, as Yann pointed out earlier. We can't just deliver the > disk cache to OpenSSL for encryption; it has to be copied into some > addressable buffer somewhere. That seems to be a major reason for the >

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/22/2017 10:34 AM, Niklas Edmundsson wrote: To make results less confusing, any specific patches/branch I should test? My baseline is httpd-2.4.25 + httpd-2.4.25-deps --with-included-apr FWIW. 2.4.25 is just fine. We'll have to make sure there's nothing substantially different about it

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Tue, 21 Feb 2017, Jacob Champion wrote: Is there interest in more real-life numbers with increasing FILE_BUCKET_BUFF_SIZE or are you already on it? Yes please! My laptop probably isn't representative of most servers; it can do nearly 3 GB/s AES-128-GCM. The more machines we test, the

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/22/2017 12:00 AM, Stefan Eissing wrote: Just so I do not misunderstand: you increased BUCKET_BUFF_SIZE in APR from 8000 to 64K? That is what you are testing? Essentially, yes, *and* turn off mmap and sendfile. My hope is to disable the mmap-optimization by default while still

Re: httpd 2.4.25, mpm_event, ssl: segfaults

> Am 22.02.2017 um 00:14 schrieb Jacob Champion : > > On 02/19/2017 01:37 PM, Niklas Edmundsson wrote: >> On Thu, 16 Feb 2017, Jacob Champion wrote: >>> So, I had already hacked my O_DIRECT bucket case to just be a copy of >>> APR's file bucket, minus the mmap() logic. I

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/19/2017 01:37 PM, Niklas Edmundsson wrote: On Thu, 16 Feb 2017, Jacob Champion wrote: So, I had already hacked my O_DIRECT bucket case to just be a copy of APR's file bucket, minus the mmap() logic. I tried making this change on top of it... ...and holy crap, for regular HTTP it's

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Mon, 20 Feb 2017, Yann Ylavic wrote: On Sun, Feb 19, 2017 at 10:11 PM, Niklas Edmundsson wrote: On Thu, 16 Feb 2017, Yann Ylavic wrote: Here I am, localhost still, 21GB file (client wget -qO- [url] &>/dev/null). Output attached. Looks good with nice big writes if I

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Sun, Feb 19, 2017 at 10:11 PM, Niklas Edmundsson wrote: > On Thu, 16 Feb 2017, Yann Ylavic wrote: >> >> Here I am, localhost still, 21GB file (client wget -qO- [url] >> &>/dev/null). >> Output attached. > > Looks good with nice big writes if I interpret it correctly. > > Is

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 16 Feb 2017, Jacob Champion wrote: On 02/16/2017 02:49 AM, Yann Ylavic wrote: +#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) /* > APR_BUCKET_BUFF_SIZE */ So, I had already hacked my O_DIRECT bucket case to just be a copy of APR's file bucket, minus the mmap() logic. I tried making

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 16 Feb 2017, Yann Ylavic wrote: Outputs (and the patch to produce them) attached. TL;DR: - http + EnableMMap=> single write - http + !EnableMMap + EnableSendfile => single write - http + !EnableMMap + !EnableSendfile => 125KB writes - https + EnableMMap

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Feb 17, 2017 2:52 PM, "William A Rowe Jr" wrote: On Feb 17, 2017 1:02 PM, "Jacob Champion" wrote: `EnableMMAP on` appears to boost performance for static files, yes, but is that because of mmap() itself, or because our bucket brigades configure

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Feb 17, 2017 1:02 PM, "Jacob Champion" wrote: `EnableMMAP on` appears to boost performance for static files, yes, but is that because of mmap() itself, or because our bucket brigades configure themselves more optimally in the mmap() code path? Yann's research is starting

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/17/2017 07:04 AM, Daniel Lescohier wrote: Is the high-level issue that: for serving static content over HTTP, you can use sendfile() from the OS filesystem cache, avoiding extra userspace copying; but if it's SSL, or any other dynamic filtering of content, you have to do extra work in

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Is the high-level issue that: for serving static content over HTTP, you can use sendfile() from the OS filesystem cache, avoiding extra userspace copying; but if it's SSL, or any other dynamic filtering of content, you have to do extra work in userspace? On Thu, Feb 16, 2017 at 6:01 PM, Yann

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 10:51 PM, Jacob Champion wrote: > On 02/16/2017 02:49 AM, Yann Ylavic wrote: >> >> +#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) /* > APR_BUCKET_BUFF_SIZE >> */ > > > So, I had already hacked my O_DIRECT bucket case to just be a copy of APR's > file

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/16/2017 02:49 AM, Yann Ylavic wrote: +#define FILE_BUCKET_BUFF_SIZE (64 * 1024 - 64) /* > APR_BUCKET_BUFF_SIZE */ So, I had already hacked my O_DIRECT bucket case to just be a copy of APR's file bucket, minus the mmap() logic. I tried making this change on top of it... ...and holy

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/16/2017 02:48 AM, Niklas Edmundsson wrote: While I applaud the efforts to get https to behave performance-wise I would hate for http to be left out of being able to do top-notch on latest networking :-) My intent in focusing there was to discover why disabling mmap() seemed to be

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/16/2017 03:41 AM, Yann Ylavic wrote: I can't reproduce it anymore, somehow I failed with my restarts between EnableMMap on=>off. Sorry for the noise... This is suspiciously similar to what I've been fighting the last three days. It's still entirely possible that you and I both messed up

Re: httpd 2.4.25, mpm_event, ssl: segfaults

dev@httpd.apache.org> >> Betreff: Re: httpd 2.4.25, mpm_event, ssl: segfaults >> >> On Thu, Feb 16, 2017 at 11:20 AM, Plüm, Rüdiger, Vodafone Group >> <ruediger.pl...@vodafone.com> wrote: >> >> >> >> Please note that "EnableMMap on" avoi

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 11:48 AM, Niklas Edmundsson wrote: > On Thu, 16 Feb 2017, Yann Ylavic wrote: > >> Here are some SSL/core_write outputs (sizes) for me, with 2.4.x. >> This is with a GET for a 2MB file, on localhost... >> >> Please note that "EnableMMap on" avoids

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 11:48 AM, Niklas Edmundsson wrote: > On Thu, 16 Feb 2017, Yann Ylavic wrote: > >> Here are some SSL/core_write outputs (sizes) for me, with 2.4.x. >> This is with a GET for a 2MB file, on localhost... >> >> Please note that "EnableMMap on" avoids

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 11:01 AM, Yann Ylavic wrote: > On Thu, Feb 16, 2017 at 10:49 AM, Yann Ylavic wrote: >> >> - http + !EnableMMap + !EnableSendfile => 125KB writes > > This is due to MAX_IOVEC_TO_WRITE being 16 in > send_brigade_nonblocking(),

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 16 Feb 2017, Yann Ylavic wrote: Here are some SSL/core_write outputs (sizes) for me, with 2.4.x. This is with a GET for a 2MB file, on localhost... Please note that "EnableMMap on" avoids EnableSendfile (i.e. "EnableMMap on" => "EnableSendfile off"), which is relevant only in the http

AW: httpd 2.4.25, mpm_event, ssl: segfaults

> -Ursprüngliche Nachricht- > Von: Yann Ylavic [mailto:ylavic@gmail.com] > Gesendet: Donnerstag, 16. Februar 2017 11:35 > An: httpd-dev <dev@httpd.apache.org> > Betreff: Re: httpd 2.4.25, mpm_event, ssl: segfaults > > On Thu, Feb 16, 2017 at 11:20 AM, Pl

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 11:20 AM, Plüm, Rüdiger, Vodafone Group wrote: >> >> Please note that "EnableMMap on" avoids EnableSendfile (i.e. >> "EnableMMap on" => "EnableSendfile off") > > Just for clarification: If you placed EnableMMap on in your test > configuration

AW: httpd 2.4.25, mpm_event, ssl: segfaults

> -Ursprüngliche Nachricht- > Von: Yann Ylavic [mailto:ylavic@gmail.com] > Gesendet: Donnerstag, 16. Februar 2017 10:49 > An: httpd-dev <dev@httpd.apache.org> > Betreff: Re: httpd 2.4.25, mpm_event, ssl: segfaults > > Here are some SSL/core_write outputs

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 10:49 AM, Yann Ylavic wrote: > > - http + !EnableMMap + !EnableSendfile => 125KB writes This is due to MAX_IOVEC_TO_WRITE being 16 in send_brigade_nonblocking(), 125KB is 16 * 8000B. So playing with MAX_IOVEC_TO_WRITE might also be worth a try for

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Here are some SSL/core_write outputs (sizes) for me, with 2.4.x. This is with a GET for a 2MB file, on localhost... Please note that "EnableMMap on" avoids EnableSendfile (i.e. "EnableMMap on" => "EnableSendfile off"), which is relevant only in the http (non-ssl) case anyway. Outputs (and the

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Not at my comp, but the mod_http2 output has special handling for file buckts. Because apr_buckt_read returns a max of 8k and splits itself. It instead grabs the file and reads the size it needs, if memory serves me well. I assume when it's mmapped it does not make much of a difference. > Am

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 12:31 AM, Yann Ylavic wrote: > > Actually this is 16K (the maximum size of a TLS record) ... these are the outputs (records) splitted/produced by SSL_write() when given inputs (plain text) greater than 16K (at once).

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 16, 2017 at 12:06 AM, Jacob Champion wrote: > On 02/15/2017 02:03 PM, Yann Ylavic wrote: > >> Assuming so :) there is also the fact that mod_ssl will encrypt/pass >> 8K buckets at a time, while the core output filter tries to send the >> whole mmap()ed file,

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/15/2017 02:03 PM, Yann Ylavic wrote: On Wed, Feb 15, 2017 at 9:50 PM, Jacob Champion wrote: For the next step, I want to find out why TLS connections see such a big performance hit when I switch off mmap(), but unencrypted connections don't... it's such a huge

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, Feb 15, 2017 at 9:50 PM, Jacob Champion wrote: > > For the next step, I want to find out why TLS connections see such a big > performance hit when I switch off mmap(), but unencrypted connections > don't... it's such a huge difference that I feel like I must be

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/07/2017 02:32 AM, Niklas Edmundsson wrote: O_DIRECT also bypasses any read-ahead logic, so you'll have to do nice and big IO etc to get good performance. Yep, confirmed... my naive approach to O_DIRECT, which reads from the file in the 8K chunks we're used to from the file bucket

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Here is how cache page replacement is done in Linux: https://linux-mm.org/PageReplacementDesign On Tue, Feb 7, 2017 at 5:32 AM, Niklas Edmundsson wrote: > On Mon, 6 Feb 2017, Jacob Champion wrote: > > > > Considering the massive amount of caching that's built into the entire

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Mon, 6 Feb 2017, Jacob Champion wrote: Considering the massive amount of caching that's built into the entire HTTP ecosystem already, O_DIRECT *might* be an effective way to do that (in which we give up filesystem optimizations and caching in return for a DMA into userspace). I have a

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/03/2017 12:30 AM, Niklas Edmundsson wrote: Methinks this makes mmap+ssl a VERY bad combination if the thing SIGBUS:es due to a simple IO error, I'll proceed with disabling mmap and see if that is a viable way to go for our workload... (Pulling from a parallel conversation, with

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Mon, Feb 6, 2017 at 12:10 PM, Ruediger Pluem wrote: >> >> What might happen in ssl_io_filter_output() is that buffered >> output data (already deleted but not cleared) end up being reused >> on shutdown. >> >> Could you please try the attached patch? > > Why would we need to

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/02/2017 11:04 AM, Yann Ylavic wrote: > Hi Niklas, > > On Wed, Feb 1, 2017 at 7:02 PM, Niklas Edmundsson wrote: >> >> We've started to see spurious segfaults with httpd 2.4.25, mpm_event, ssl on >> Ubuntu 14.04LTS. Not frequent, but none the less happening. >> >> #4

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Hmm, Linux raises SIGBUS if an mmap is used after the underlying file has been truncated (see [1]). See also https://bz.apache.org/bugzilla/show_bug.cgi?id=46688 . Niklas, just to clarify: you're not willfully truncating large files as they're being served, right? I *can* reproduce a SIGBUS

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 2 Feb 2017, Jacob Champion wrote: We've started to see spurious segfaults with httpd 2.4.25, mpm_event, ssl on Ubuntu 14.04LTS. Not frequent, but none the less happening. #4 ssl_io_filter_output (f=0x7f507013cfe0, bb=0x7f4f840be168) at ssl_engine_io.c:1746 data =

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 2 Feb 2017, Jacob Champion wrote: On 02/02/2017 03:05 PM, Yann Ylavic wrote: Hmm, Linux raises SIGBUS if an mmap is used after the underlying file has been truncated (see [1]). See also https://bz.apache.org/bugzilla/show_bug.cgi?id=46688 . Niklas, just to clarify: you're not

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/02/2017 03:05 PM, Yann Ylavic wrote: Couldn't htcacheclean or alike do something like this? "EnableMMAP off" could definitely help here. (Didn't mean to ignore this part of your email, but I don't have much experience with htcacheclean yet so I can't really comment...) --Jacob

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/02/2017 03:05 PM, Yann Ylavic wrote: Hmm, Linux raises SIGBUS if an mmap is used after the underlying file has been truncated (see [1]). See also https://bz.apache.org/bugzilla/show_bug.cgi?id=46688 . Niklas, just to clarify: you're not willfully truncating large files as they're being

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 2, 2017 at 11:36 PM, Jacob Champion wrote: > On 02/02/2017 02:32 PM, Yann Ylavic wrote: >> >> On Thu, Feb 2, 2017 at 11:19 PM, Jacob Champion >> wrote: >>> >>> Idle thoughts: "Cannot access memory" in this case could be a red >>> herring,

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/02/2017 02:32 PM, Yann Ylavic wrote: On Thu, Feb 2, 2017 at 11:19 PM, Jacob Champion wrote: Idle thoughts: "Cannot access memory" in this case could be a red herring, if Niklas' gdb can't peer into mmap'd memory spaces [1]. It seems reasonable that the data in

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, Feb 2, 2017 at 11:19 PM, Jacob Champion wrote: > > Idle thoughts: "Cannot access memory" in this case could be a red herring, > if Niklas' gdb can't peer into mmap'd memory spaces [1]. It seems reasonable > that the data in question could be mmap'd, given the nice

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On 02/02/2017 02:04 AM, Yann Ylavic wrote: Hi Niklas, On Wed, Feb 1, 2017 at 7:02 PM, Niklas Edmundsson wrote: We've started to see spurious segfaults with httpd 2.4.25, mpm_event, ssl on Ubuntu 14.04LTS. Not frequent, but none the less happening. #4 ssl_io_filter_output

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 2 Feb 2017, Niklas Edmundsson wrote: On Thu, 2 Feb 2017, Yann Ylavic wrote: Are we hitting a corner case of process cleanup that plays merry hell with https/ssl, or are we just having bad luck? Ideas? Suggestions? 2.4.25 is eager to terminate/shutdown keepalive connections more

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Thu, 2 Feb 2017, Yann Ylavic wrote: Are we hitting a corner case of process cleanup that plays merry hell with https/ssl, or are we just having bad luck? Ideas? Suggestions? 2.4.25 is eager to terminate/shutdown keepalive connections more quickly (than previous versions) on graceful

Re: httpd 2.4.25, mpm_event, ssl: segfaults

Hi Niklas, On Wed, Feb 1, 2017 at 7:02 PM, Niklas Edmundsson wrote: > > We've started to see spurious segfaults with httpd 2.4.25, mpm_event, ssl on > Ubuntu 14.04LTS. Not frequent, but none the less happening. > > #4 ssl_io_filter_output (f=0x7f507013cfe0, bb=0x7f4f840be168)

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, 1 Feb 2017, Eric Covener wrote: On Wed, Feb 1, 2017 at 1:02 PM, Niklas Edmundsson wrote: This might be due to processes being cleaned up due to hitting MaxSpareThreads or MaxConnectionsPerChild, these are tuned to not happen frequently. It's just a wild guess, but

Re: httpd 2.4.25, mpm_event, ssl: segfaults

On Wed, Feb 1, 2017 at 1:02 PM, Niklas Edmundsson wrote: > This might be due to processes being cleaned up due to hitting > MaxSpareThreads or MaxConnectionsPerChild, these are tuned to not happen > frequently. It's just a wild guess, but the reason for me suspecting this is >

httpd 2.4.25, mpm_event, ssl: segfaults

Hi all! We've started to see spurious segfaults with httpd 2.4.25, mpm_event, ssl on Ubuntu 14.04LTS. Not frequent, but none the less happening. This might be due to processes being cleaned up due to hitting MaxSpareThreads or MaxConnectionsPerChild, these are tuned to not happen