> Is it possible to disable some or all optimizations for a certain user > agent (crawler)? When it crawls the site, CPU usage for the nginx process is > getting too high primarily because of the pagespeed module.
That would be for your general configuration. Pagespeed is a module, not a complete replacement for your server! If a crawler is so ill-behaved as to flood your server, I'd simply ban it. Have the server return an instant 400. Rules for well-behaved robots have been perfectly clear for more than twenty years. @pagespeed - this isn't even a development issue, yet like everything else it arrived through github. Is there really anything happening at apache, as opposed to being a purely github project? -- Nick Kew
