Re: HTTP/2 in massive file transfers

That's a great benchmark, José. A couple of questions: 1. When using http 1.1, is your connection over SSL? I assume that your h2 connection is over ssl, and that might account for a speed penalty. 2. Have you tried this on other h2 implementations (h2o, nginx)? 3. Can you take some other data po

Re: Sharing information across Apache child processes

There's a variation on this theme you might consider. You can use a few counters in shared memory, and save config information in a file. If one of the child processes learns of a config update (say by handling an HTTP request for one from somewhere) it can update the file (atomically, say, via w

Re: sequence of request

Yes, in particular, the gdb feature that can help determine who is changing r->filename is hardware watchpoints: http://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.html 1. Set a breakpoint at a line of code where you believe r->filename is correct 2. Start up Apache with "run -X" 3. Issue a

Re: associating an object with a request

Thanks Eric. That looks perfect! I'll give it a try. On Wed, Apr 23, 2014 at 4:34 PM, Eric Covener wrote: > On Wed, Apr 23, 2014 at 4:31 PM, Joshua Marantz > wrote: > > > > I want to create a data structure and associate it with a request_rec*, > > which will be

associating an object with a request

Hi, I want to create a data structure and associate it with a request_rec*, which will be accessed by multiple filters and handlers in our module. I know I can use the filter->ctx for general data structure storage, but I actually want something that will be accessed by multiple filters and handl

Re: problem with different version of openssl in Apache and apache module

We faced this exact issue (openssl clashes with other linked-in versions) in mod_pagespeed and ngx_pagespeed, its nginx equivalent. We solved this problem in our Apache module because we linked mod_pagespeed.so hiding all the symbols other than the module entry-point into Apache. Here's the link

Re: Apache Buckets and Brigade

On Wed, May 1, 2013 at 12:14 PM, Sindhi Sindhi wrote: > Thanks to all for the reply. > > Josh, the concern I mentioned was, we may not want mod_pagespeed to modify > the in-memory HTML content. The only change we may want to see in our HTML > will be that the old strings are replaced by the new s

Re: Apache Buckets and Brigade

in a conf file. -Josh On Wed, May 1, 2013 at 11:19 AM, Jim Jagielski wrote: > How is that different from mod_substitute and/or mod_sed? > > On May 1, 2013, at 9:22 AM, Joshua Marantz wrote: > > > I have a crazy idea for you. Maybe this is overkill but this sounds like &

Re: Apache Buckets and Brigade

On Wed, May 1, 2013 at 9:48 AM, Sindhi Sindhi wrote: > I doubt if I can use ModPagespeedSubstitute, because > our string replacement actually uses some business logic. For ex. in > "oldString", if i find a "old" string at offset 0 i'll replace it with > I agree: that configuration-only solution

Re: Apache Buckets and Brigade

I have a crazy idea for you. Maybe this is overkill but this sounds like it'd be natural to add to mod_pagespeed as a new filter. Here's some code you might use as a template https://code.google.com/p/modpagespeed/source/browse/trunk/src/net/instaweb/rewriter/collapse_w

Re: apr_memcache operation timeouts

013 at 10:45 AM, Jeff Trawick wrote: > On Tue, Mar 12, 2013 at 1:56 PM, Jeff Trawick wrote: > > On Mon, Mar 11, 2013 at 3:50 PM, Joshua Marantz > wrote: > >> > >> ping! > >> > >> Please don't hesitate to push back and tell me if I can sup

Re: apr_memcache operation timeouts

rstand what the testing flow is for APR. I'd be happy to add unit-tests for that if someone points me to a change-list or patch-file that does it properly. -Josh On Thu, Nov 1, 2012 at 8:04 AM, Joshua Marantz wrote: > I have completed a solution to this problem, which can be a drop-in up

Re: apr_memcache operation timeouts

the strategy I used to solve this problem, then please let me know if I can help with any changes required to get this into the main distribution, On Mon, Oct 22, 2012 at 5:21 PM, Joshua Marantz wrote: > I've had some preliminary success with my own variant of apr_memcache.c > (creativ

When does Apache restart child processes

Hi, While load-testing our module, I've noticed (via log statements) that child processes get frequently shut down & restarted. I'd like to figure out why. I know that Apache can be configured to restart child processes periodically based on number of requests served, but I don't believe that's

Re: apr_memcache operation timeouts

her than relying on the socket timeout. The socket timeout didn't work as well as this though. Does anyone have any theories as to why, or what could be done to the patch in https://issues.apache.org/bugzilla/show_bug.cgi?id=51065 to work? -Josh On Fri, Oct 19, 2012 at 9:25 AM, Joshua Mar

Forcing Apache to exit on startup

Hi, Our module has multiple confirmation parameters. There is a case where if you have option A and option B, then you must also specify option C, otherwise Bad things can happen that are a lot easier to debug on startup than they are after the server is running. I know how to force 'apachectl'

Re: apr_memcache operation timeouts

chanism of timing out the calls from my side. -Josh On Thu, Oct 18, 2012 at 10:46 AM, Joshua Marantz wrote: > Thanks Jeff, that is very helpful. We are considering a course of action > and before doing any work toward this, I'd like to understand the pitfalls > from people that

Re: apr_memcache operation timeouts

nd to discard the connection if the timeout is reached". That sounds like a pattern that might be found elsewhere in the Apache HTTPD code base. Thanks in advance for your help! -Josh On Wed, Oct 17, 2012 at 8:16 PM, Jeff Trawick wrote: > On Wed, Oct 17, 2012 at 3:36 PM, Joshua Marantz

apr_memcache operation timeouts

Is there a mechanism to time out individual operations? If memcached freezes, then it appears my calls to 'get' will block until memcached wakes up. Is there any way to set a timeout for that call? I can repro this in my unit tests by sending a SIGSTOP to memcached before doing a 'get'? Here ar

Re: aprmemcache question

d wakes up. Is there any way to set a timeout for that call? I can repro this in my unit tests by sending a SIGSTOP to memcached before doing a 'get'. -Josh On Thu, Sep 27, 2012 at 4:37 PM, Joshua Marantz wrote: > This helps a lot. I think 600 seconds seems like a fine idle

Re: aprmemcache question

at 1:55 PM, Joshua Marantz > wrote: > > That one call-site is HTTP_24/src/modules/cache/mod_socache_memcache.c, > > right? That was where I stole my args from. > > no, subversion > > > As the TCP/IP layer is a lower level abstraction than bathe apr_memcache > &g

Re: aprmemcache question

AM, Jeff Trawick wrote: > On Thu, Sep 27, 2012 at 11:15 AM, Joshua Marantz > wrote: > > On Thu, Sep 27, 2012 at 10:58 AM, Ben Noordhuis > wrote: > >> > >> If dlsym() is called with the special handle NULL, it is interpreted > as > >> a >

Re: aprmemcache question

On Thu, Sep 27, 2012 at 10:58 AM, Ben Noordhuis wrote: > If dlsym() is called with the special handle NULL, it is interpreted as a > reference to the executable or shared object from which the call is being > made. Thus a shared object can reference its own symbols. > > And that's how it w

Re: aprmemcache question

case. Do you have a feel for the exact meaning of that TTL parameter to apr_memcache_server_create? -Josh On Thu, Sep 27, 2012 at 8:53 AM, Ben Noordhuis wrote: > On Thu, Sep 27, 2012 at 4:05 AM, Joshua Marantz > wrote: > > RE "failing the build of my module" -- the

Re: aprmemcache question

12 at 7:31 PM, Joshua Marantz > wrote: > > Looking at source, I see that Jeff's patch, and the 'ttl' parameter in > > general, is only referenced under '#if APR_HAS_THREADS'. When I > > load-tested and found the timeouts, I was testing under Apach

Re: aprmemcache question

sh On Wed, Sep 26, 2012 at 7:17 PM, Joshua Marantz wrote: > +dev (sorry for the duplicate; my first attempt failed due to not being a > subscriber). > > Keeping modules-dev on CC if that's appropriate. > > Thanks, Jeff, I was wondering if there was a units issue there.

Re: aprmemcache question

lling this function for module developers? -Josh On Wed, Sep 26, 2012 at 6:20 PM, Jeff Trawick wrote: > On Wed, Sep 26, 2012 at 5:38 PM, Joshua Marantz > wrote: > > Hi, > > > > I've been having some success with the apr_memcache_* functions. In > > load-tests, howe

Re: aprmemcache question

Jeff Trawick wrote: > On Wed, Sep 26, 2012 at 5:38 PM, Joshua Marantz > wrote: > > Hi, > > > > I've been having some success with the apr_memcache_* functions. In > > load-tests, however, I'm finding a lot of timeouts > > with apr_memcache_mult

aprmemcache question

Hi, I've been having some success with the apr_memcache_* functions. In load-tests, however, I'm finding a lot of timeouts with apr_memcache_multgetp. Specifically, the status returned with the individual elements is APR_TIMEUP. This leads me to wonder what the significance of the second to las

Re: apr_memcache vs libmemcached

e better than synchronous reads form disk. Thanks, -Josh On Thu, Jul 19, 2012 at 5:39 PM, Nick Kew wrote: > On Thu, 19 Jul 2012 16:41:51 -0400 > Joshua Marantz wrote: > > > Hi, > > > > Is anyone maintaining apr_memcache? Or should I stick with libmemcached? > &g

apr_memcache vs libmemcached

Hi, Is anyone maintaining apr_memcache? Or should I stick with libmemcached? This page: http://code.google.com/p/memcached/wiki/Clients suggests apr_memcached is not maintained. Thanks! -Josh

Re: Get the directory of the module

Modules register a configuration callback called on every declared option. So they don't need to know where the configuration file is; they get called with the parsed config-file options. See http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/examples/mod_example_hooks.cfunction cmd_example

Re: Get the directory of the module

e able to tell his module where the files are. > > > On 6/11/2012 7:40 PM, Joshua Marantz wrote: > >> I think he wants to write bytecodes during module installation when he >> will >> have write access. >> >> During installation can you also write a .con

Re: Get the directory of the module

I think he wants to write bytecodes during module installation when he will have write access. During installation can you also write a .conf template showing your module where the bytecodes are? Josh On Jun 11, 2012 7:29 PM, "Jerry Stuckle" wrote: > > > On 6/11/2012 5:49 PM, Ben Noordhuis wro

Re: Determining mimetype of document

h On Fri, Jun 8, 2012 at 5:16 PM, Joe Lewis wrote: > On 06/08/2012 03:07 PM, Joshua Marantz wrote: > >> Hi, >> >> I'm trying to figure out whether my filter, which currently runs upstream >> of mod_headers, is likely to see the correct mimetype >> in reque

Determining mimetype of document

Hi, I'm trying to figure out whether my filter, which currently runs upstream of mod_headers, is likely to see the correct mimetype in request->content_type. In particular, we need to know whether browsers will interpret the content as XHTML or HTML, which is determined by the mimetype. Do peopl

Re: output filter ordering question: mod_deflate vs mod_headers vs mod_pagespeed

the correct way to look at this data? In any case the evidence from my system-tests suggest that I haven't achieved the ordering I want yet. Is there something else I need to do? On Thu, May 17, 2012 at 9:27 AM, Joshua Marantz wrote: > Thanks for the quick reply, Nick. I played around

Re: output filter ordering question: mod_deflate vs mod_headers vs mod_pagespeed

elaborate on the state of mod_filter? Thanks! -Josh On Thu, May 17, 2012 at 2:30 AM, Nick Kew wrote: > > On 17 May 2012, at 04:24, Joshua Marantz wrote: > > > Or is that insane &/or dangerous? > > AP_FTYPE_ values are something of a blunt instrument, not ideal > for case

output filter ordering question: mod_deflate vs mod_headers vs mod_pagespeed

Hi, Currently, mod_pagespeed runs before mod_headers by registering as: ap_register_output_filter( kModPagespeedFilterName, instaweb_out_filter, NULL, static_cast(AP_FTYPE_RESOURCE + 1)); mod_headers registers with: ap_register_output_f

Re: STL/Boost containers in Apache module

Yes, absolutely. mod_pagespeed uses STL extensively. All required support libraries and linking magic gets pulled into the .so. We don't use Boost but I suspect that would work fine as well, though you might get quite a large .so as a result of linking it in. -Josh On Mon, Apr 23, 2012 at 8:2

Re: Some fundamental questions about output filters

Every new module-implementor should look at this example: http://svn.apache.org/repos/asf/httpd/httpd/trunk/modules/examples/mod_example_hooks.c Although I'm not particularly proud of how complicated our apache-gasket has gotten -- it could use a refactor or 5, this is mod_pagespeed's output filt

Re: Inject html into response body

What's your thinking on how you are going to parse the HTML to inject the code? Do you need to parse arbitrary HTML from any website or is this filter targeted at a particular site whose HTML is tailored for this purpose? -Josh On Sun, Dec 4, 2011 at 5:15 PM, wrote: > On Sun, Dec 04, 2011 at 1

Re: Question about malloc / realloc in module

If you malloc/calloc/realloc without a free you will leak memory. Do you have some reason to believe that another module might prevent your module's calls to free() from being run? What do you have in mind specifically? You can also mimic realloc by just allocating more memory from the pool and

Detecting which MPM a module is running in

Hello from mod_pagespeed again. We are adding support for running in the Worker MPM, having spent most of our time since we launched the product sheltered in the prefork MPM where our multi-threading challenges are all of our own making. Having tuned our threading model for prefork, where all req

po

Hi, I've been load-testing our module (mod_pagespeed) with httpd 2.2.16 built with these options: --enable-pool-debug --with-mpm=worker I've been getting periodic aborts from apr_table_addn that don't look like they are from my module.

Re: Vary:User-Agent, best practices, and making the web faster.

phase in vary:user-agent compliance without falling off a performance cliff. -Josh On Sun, Jun 5, 2011 at 2:54 PM, Ben Noordhuis wrote: > On Sun, Jun 5, 2011 at 13:42, Joshua Marantz wrote: > > This is a case where the content varies based on user-agent. The > > recommendation on th

Re: Vary:User-Agent, best practices, and making the web faster.

On Sun, Jun 5, 2011 at 7:32 AM, Ben Noordhuis wrote: > On Sun, Jun 5, 2011 at 02:15, Joshua Marantz wrote: > > On Sat, Jun 4, 2011 at 7:58 PM, Ben Noordhuis > wrote: > >> Some popular OSS packages depend on Vary: User-Agent to make > >> downstream proxies (re

Re: Vary:User-Agent, best practices, and making the web faster.

On Sat, Jun 4, 2011 at 7:58 PM, Ben Noordhuis wrote: > > And I still don't understand how that relates to Vary:User-Agent. > What's > > really at issue here seems more related to proxies; is that right? That > > proxies were not respecting Accept-Encoding, but sending gzipped content > to > >

Re: Vary:User-Agent, best practices, and making the web faster.

M, Ben Noordhuis wrote: > On Sat, Jun 4, 2011 at 21:26, Joshua Marantz wrote: > > I think what we'd do is basically let mod_pagespeed ignore > "Vary:User-Agent" > > if we saw that it was inserted per this exact pattern. This would, to be > > This seems like a stu

Vary:User-Agent, best practices, and making the web faster.

Hi, We've been working on a lingering HTTP-compliance issue in mod_pagespeed: respecting Vary:User-Agent. mod_pagespeed needs to cache resources in order to optimize them. The economics of this make sense when the server optimizes a resource, and saves the optimization for serving to multiple cl

Re: KeepAlive -- why is it off by default?

age refresh had approximately 7-second stairsteps -- I think 2 seconds for linger and 5 seconds for KeepAlive. -Josh On Mon, Apr 11, 2011 at 9:29 PM, Ben Noordhuis wrote: > On Tue, Apr 12, 2011 at 02:18, Joshua Marantz wrote: > > What are the reasons someone might wish to turn KeepAlive of

KeepAlive -- why is it off by default?

webpagetest.org gives a big red *F *to web sites that do* *not have KeepAlive enabled. Yet, at least in the CentOS installation of Apache, it is off by default. Why is that? What are the reasons that KeepAlive should be off? Although it's trivial to turn it on in httpd.conf, many web sites do n

Re: how to parse html content in handler

mod_pagespeed's event-driven HTML parser is open source, and is written in C++: http://code.google.com/p/modpagespeed/source/browse/trunk/src/net/instaweb/htmlparse/public/html_parse.h This pa

Re: ordering output filters

Thanks, Nick & Ben! On Mon, Mar 14, 2011 at 1:31 PM, Nick Kew wrote: > > AP_FTYPE_RESOURCE+1. That also leaves an admin the possibility of > overriding it. I didn't realize these +1/-1 hacks were available for this API. This looks really simple & is the direction I'm leaning. > Why not an i

ordering output filters

Hello from mod_pagespeed again. Our users have identified another incompatibility between standard filters and mod_pagespeed; this time with mod_includes. In general I think that mod_pagespeed should run after mod_includes, for a few reasons. But in particular, mod_pagespeed, in its own html-ce

Re: Saving the original request URI ahead of a mod_rewrite

This is great data, Ben & Eric. I'm a little worried about the "follow until" in terms of performance (probably not really an issue most of the time) and vulnerability to other modules corrupting state (for which mod_pagespeed would be blamed loudly on twitter :) So I'm still thinking of the usin

Re: Saving the original request URI ahead of a mod_rewrite

A more detailed analysis of what happens in Apache for this testcase is here: http://code.google.com/p/modpagespeed/issues/detail?id=234#c16 The code says "internal_internal_redirect" but it makes a new request. I haven't had a chanc

Saving the original request URI ahead of a mod_rewrite

Hi, A new bug has surfaced in mod_pagespeed that we understand, but would welcome advice on the best way to fix. The problem is tracked in http://code.google.com/p/modpagespeed/issues/detail?id=234. Briefly, mod_pagespeedseeks to impr

Re: Help trying to figure out why an output_filter is not called.

site security: > http://www.bettercgi.com/strongbox/ > > Throttlebox - Intelligent Bandwidth Control > http://www.bettercgi.com/throttlebox/ > > Strongbox / Throttlebox affiliate program: > http://www.bettercgi.com/affiliates/user/register.php > > > > On 01/05/2011 0

Re: Help trying to figure out why an output_filter is not called.

Right you are. That's much simpler then. Thanks! -Josh On Wed, Jan 5, 2011 at 3:46 PM, Ben Noordhuis wrote: > On Wed, Jan 5, 2011 at 20:40, Joshua Marantz wrote: > > So if I try to remove the 'expires' filter from my handler (which runs > > early) then mod_ex

Re: Help trying to figure out why an output_filter is not called.

On Wed, Jan 5, 2011 at 10:43 AM, Ben Noordhuis wrote: > > I guess we should eliminate FIXUP_HEADERS_OUT, FIXUP_HEADERS_ERR, and > > MOD_EXPIRES. > Are there any other similar header-mucking-filters I need to kill? > Moreover, expires_insert_filter runs as APR_HOOK_MIDDLE which means it > run

Re: Help trying to figure out why an output_filter is not called.

ick Kew wrote: > On Wed, 5 Jan 2011 08:45:57 -0500 > Joshua Marantz wrote: > > > > What might be going wrong in his server to cause this to fail? Could > some > > other filter be somehow finding our filter and killing it? Or sending > the > > bytes directly

Re: Help trying to figure out why an output_filter is not called.

Thanks again for the fast response, Ben! On Wed, Jan 5, 2011 at 8:57 AM, Ben Noordhuis wrote: > On Wed, Jan 5, 2011 at 14:45, Joshua Marantz wrote: > > other filter be somehow finding our filter and killing it? Or sending > the > > bytes directly to the network before our f

Help trying to figure out why an output_filter is not called.

One of the improvements mod_pagespeed is supposed to do to sites is extend the cache lifetime of their resources indefinitely by including a content hash in the URL. This is working well for a large number of sites, but I encountered one today where it does not work. To accomplish the cache exten

Re: Overriding mod_rewrite from another module

On Mon, Jan 3, 2011 at 6:15 PM, Eric Covener wrote: > >> The access checking on mod_pagespeed resources is > >> redundant, because the resource will either be served from cache (in > which > >> case it had to be authenticated to get into the cache in the first > place) or > >> will be decoded and

Re: Overriding mod_rewrite from another module

On Mon, Jan 3, 2011 at 4:50 PM, Ben Noordhuis wrote: > > This means that returning OK from my handler does not prevent > > mod_authz_host's handler from being called. > > You're mistaken, Joshua. The access_checker hook by default is empty. > mod_authz_host is a module and it can be disabled (if

Re: Overriding mod_rewrite from another module

e same server with full authentication. I'd appreciate any comments on this approach. -Josh On Mon, Jan 3, 2011 at 11:40 AM, Joshua Marantz wrote: > OK I tried to find a more robust alternative but could not. I was thinking > I could duplicate whatever mod_rewrite was doing to se

Re: Overriding mod_rewrite from another module

). Is there some magic setting we could put into the request structure to tell Apache not to interpret the request as being mapped from a file, but just to pass it through to our handler? Thanks! -Josh On Sat, Jan 1, 2011 at 6:24 AM, Ben Noordhuis wrote: > On Sat, Jan 1, 2011 at 00:16, Jos

Re: Overriding mod_rewrite from another module

.amazon.com/Apache-Modules-Book-Application-Development/dp/0132409674/ref=sr_1_1?ie=UTF8&qid=1293837117&sr=8-1>book offer much detail. code.google.com fares a little better but just points to 4 existing usages. -Josh On Fri, Dec 31, 2010 at 1:50 PM, Ben Noordhuis wrote: > On Fri, Dec 3

Overriding mod_rewrite from another module

I need to find the best way to prevent mod_rewrite from renaming resources that are generated by a different module, specifically mod_pagespeed. This needs to be done from within mod_pagespeed, rather than asking the site admin to tweak his rule set. By reading mod_rewrite.c, I found a mechanism

Re: Howto unittest httpd modules?

For mod_pagespeed we do 2 of the 3 testing modes discussed. We have the majority of our code running independent of Apache so it can be aggressively unit-tested with googletest . We also have automated