Using forward proxy to continue downloading a Range request using mod_disk_cache
Hello All, I am testing with an Apache server 2.2.9 in forward proxy mode (ProxyRequests On). Here is what I am trying to do: * When a client issues a Range request, requesting say 0-x bytes, I would satisfy that request up from the origin server, but would like to continue downloading the response from the origin server and store it in the cache. So that any future range request for the same object would go from the cache. * I am thinking of writing an output_filter (possibly another cache provider for store_body) which will be added when such a request comes in. Another module will strip out the Range headers and forward the request to the origin server. On the response path, continue the processing just like mod_disk_cache's store_body. When that range request is satisfied it will create the EOS bucket in a brigade and pass it along to the core_output_filter, which will then send it to the client and do the essential close routines. The rest of data coming from the origin server will be written to the disk by this filter and nothing passed along to the output_filter chain. Basically I am trying to close the connection on the client-side but continue downloading from the server-side. Is this possible, or does this break the filter chains. Are there any other methods by which I can do this ? Thank you for your time. -- Pranav
Re: incubator proposal for (what was once) Inktomi Traffic Server
On Sun, Jun 14, 2009 at 9:25 PM, Roy T. Fieldingfield...@gbiv.com wrote: I think this is an interesting opportunity to compare different implementations and share code where desirable. I haven't seen anyone comment on the proposal yet. Roy Begin forwarded message: From: Leif Hedstrom l...@yahoo-inc.com Date: June 12, 2009 9:17:59 AM PDT To: gene...@incubator.apache.org Subject: [Proposal] Traffic Server Good morning, We would like to submit the Traffic Server proposal to the incubator. Our draft is available at http://wiki.apache.org/incubator/TrafficServerProposal This sounds awesome. Can it act as a forward proxy as well ? I guess we will find out once the source is released. The 35000 rps is for reverse proxy or forward proxy ? -- Pranav A quick overview of Traffic Server: Traffic Server is a Yahoo! / Inktomi caching proxy server. It has been actively developed and used inside Yahoo! for the last 3+ years, and we are now ready to begin the next step in it's evolution: make it Open Source. TS is a fairly large piece of software (300k+ lines of C/C++ code), and provides features and benefits lacking in many existing proxy/caches. I am obviously looking for feedback and comments on the proposal, as well as a few mentors. Doug Cutting has accepted to be our Champion. Thanks, -- Leif
Re: incubator proposal for (what was once) Inktomi Traffic Server
On Mon, Jun 15, 2009 at 12:06 PM, Akins, Brianbrian.ak...@turner.com wrote: On 6/15/09 12:39 PM, Pranav Desai pranavade...@gmail.com wrote: The 35000 rps is for reverse proxy or forward proxy ? Of course, I've coerced a stock mod_disk_cache/mod_proxy_http httpd server to over 40k rps on normal serevr hardware. It all depends on the benchmark. I haven't done much testing with mod_proxy_http as a forward proxy, but 40k seems really good ! ... will it be possible for you share your mpm model and related parameters and what test tool do you use to generate that load ? Thanks -- Pranav FWIW, the more code we can play with, the better. Who knows what goodies we may find. -- Brian Akins
Re: Transparent proxy setup works fine, but want to confirm the settings
On Sat, Feb 14, 2009 at 1:03 PM, Ruediger Pluem rpl...@apache.org wrote: On 02/14/2009 08:59 PM, Rainer Jung wrote: If you do reverse proxy only via RewriteRule, then you end up using no connection pool (i.e. no persistent connections) to the HTTP_HOSTs. In case there are only few of those (or few that carry the most load), you would better define a connection pool to them with ProxyPass. If you want to keep your rewrite construction, you can use a URL in ProxyPass, which you know won't really occur: ProxyPass /does/not/exist http://most.important.host/ smax=... ... He is doing forward proxying here and not reverse proxying. In order to create a pool IMHO the better approach is Proxy http://most.important.host/ ProxySet smax=... /Proxy I am confused a bit here. With the RewriteRule I mentioned earlier will I lose persistent connections for transparent proxy connections ? And the above settings in addition to the RewriteRules will help in getting persistent connections ... ? -- Pranav Regards Rüdiger
Re: Transparent proxy setup works fine, but want to confirm the settings
On Fri, Feb 13, 2009 at 1:26 AM, Graham Leggett minf...@sharp.fm wrote:
Pranav Desai wrote:
I am trying to setup Apache 2.2.9 as a transparent proxy. So that the
users don't have to configure their browsers. Now the URLs coming in
are relative for transparent proxy, so normally apache tries to look
it up on the filesystem and it obviously fails. So I added a
RewriteRule to convert the relative to absolute URLs.
RewriteEngine On
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
RewriteLog logs/rewrite_log
RewriteLogLevel 5
Now, it works perfectly for all traffic expect the one that is
destined for the server itself. E.g.
http://apache_proxy_ip:port/
Whenever I access the above link, the rewrite engine loops and the
server reaches the MaxClient. I have included the log below.
That would make perfect sense though, you are asking the server to send you
to the server prefixed with the host header, and when you use the hostname
of the proxy server itself, you create a loop by definition, which means...
So, I added some conditions to not apply the RewriteRule for HOST
destined to the server.
RewriteCond %{HTTP_HOST} !10.1.0.206.*
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
...this is a sensible workaround.
I wanted to confirm if this is the right way to do transparent proxy
or is there a better way to make it more solid ?
In theory this will work as is, I am not sure whether there is an option in
the proxy to do this natively without the need for rewrite.
I checked the proxy, and there isn't anything to specifically do this,
but maybe I could have used some ReverseProxy config to get the same
behavior, but I thought RewriteRule was a bit cleaner.
-- Pranav
Regards,
Graham
--
Transparent proxy setup works fine, but want to confirm the settings
Hello All,
I am trying to setup Apache 2.2.9 as a transparent proxy. So that the
users don't have to configure their browsers. Now the URLs coming in
are relative for transparent proxy, so normally apache tries to look
it up on the filesystem and it obviously fails. So I added a
RewriteRule to convert the relative to absolute URLs.
RewriteEngine On
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
RewriteLog logs/rewrite_log
RewriteLogLevel 5
Now, it works perfectly for all traffic expect the one that is
destined for the server itself. E.g.
http://apache_proxy_ip:port/
Whenever I access the above link, the rewrite engine loops and the
server reaches the MaxClient. I have included the log below.
So, I added some conditions to not apply the RewriteRule for HOST
destined to the server.
RewriteCond %{HTTP_HOST} !10.1.0.206.*
RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [P]
I wanted to confirm if this is the right way to do transparent proxy
or is there a better way to make it more solid ?
Just to mention, I want it to act primarily like a proxy server, so
losing/blocking all webserver functionality is also fine, as long as I
get the /server-status page. But I dont want a single url to fail the
server. So I will be fine if there is a better way to get the
transparency working while affecting the webserver.
Thanks for your time.
-- Pranav
== lots of these message in the rewrite_log ===
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) init rewrite
engine with requested uri /
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (3) applying pattern
'^/(.*)' to uri '/'
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (4) RewriteCond:
input='/' pattern='!^/server.
*' = matched
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) rewrite '/' -
'http://10.1.0.206:2901/'
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) forcing
proxy-throughput with http://10.1.
0.206:2901/
10.1.0.156 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (1) go-ahead with
proxy request proxy:http://1
0.1.0.206:2901/ [OK]
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) init rewrite
engine with requested uri /
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (3) applying pattern
'^/(.*)' to uri '/'
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (4) RewriteCond:
input='/' pattern='!^/server.
*' = matched
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) rewrite '/' -
'http://10.1.0.206:2901/'
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (2) forcing
proxy-throughput with http://10.1.
0.206:2901/
10.1.0.206 - - [12/Feb/2009:17:51:09 --0800]
[10.1.0.206/sid#1a5bdab8][rid#1a6d66b8/initial] (1) go-ahead with
proxy request proxy:http://1
0.1.0.206:2901/ [OK]
Re: Need suggestions for adding tproxy support to mod_proxy
On Thu, Dec 25, 2008 at 5:52 PM, Stefan Fritsch s...@sfritsch.de wrote: Hi, On Wed, 17 Dec 2008, Pranav Desai wrote: I am trying to add tproxy4 (http://www.balabit.com/support/community/products/tproxy/) support to the mod_proxy to achieve transparency. It basically involves a kernel patch which allows binding of a socket to foreign address among other things. At the app layer we only need to set the setsockopt() Linux 2.6.28 includes tproxy support again (see Documentation/networking/tproxy.txt in the kernel source). You may want to check that your httpd patch works with that kernel, too. Yeah, I read that in one of the linux-net mailing lists, but I wasn't sure which version it was going to be. So, thanks for that. I don't think there should be any change in the patch, but I will test it with the new kernel. BTW, I think this would be a nice feature to have in httpd 2.4. Sure, sounds good. I will create a patch for that. Cheers, Stefan
Re: Need suggestions for adding tproxy support to mod_proxy
On Fri, Dec 19, 2008 at 9:06 PM, Pranav Desai pranavade...@gmail.com wrote: On Thu, Dec 18, 2008 at 2:34 AM, Graham Leggett minf...@sharp.fm wrote: Pranav Desai wrote: Yeah, the application changes are restricted to a few lines. I believe you mean the connect_backend() and not the proxy_connect module for the CONNECT method ? I did yes, sorry. If this can be made available to all the proxy modules in one go, it would be ideal. There are more changes than I thought there would be. Tproxy needs the CAP_NET_ADMIN capability for setting the setsockopt(). So it seems like I have to preserve the capabilities using prctl and then after the effective user changes to non-privileged, set the CAP_NET_ADMIN capability for that process. What I am not sure of is: * Whats the best place to keep the capabilities, since it would have to be done before it drops the privilege. * Would I have to add the capability for all processes that are created for handling requests ? Is there a better way to set the capabilities of all the spawned processes ? I have included a patch for this support. Please let me know if I have missed things. I tried to restrict the changes to the module alone, but due to the nature of the changes, I have to go into the os/ and also the srclib area. If you guys can suggest any better solution please let me know. http://miscfiles.googlecode.com/svn/trunk/tproxy.patch Thanks -- Pranav Thanks -- Pranav Regards, Graham --
Re: Need suggestions for adding tproxy support to mod_proxy
On Thu, Dec 18, 2008 at 2:34 AM, Graham Leggett minf...@sharp.fm wrote: Pranav Desai wrote: Yeah, the application changes are restricted to a few lines. I believe you mean the connect_backend() and not the proxy_connect module for the CONNECT method ? I did yes, sorry. If this can be made available to all the proxy modules in one go, it would be ideal. There are more changes than I thought there would be. Tproxy needs the CAP_NET_ADMIN capability for setting the setsockopt(). So it seems like I have to preserve the capabilities using prctl and then after the effective user changes to non-privileged, set the CAP_NET_ADMIN capability for that process. What I am not sure of is: * Whats the best place to keep the capabilities, since it would have to be done before it drops the privilege. * Would I have to add the capability for all processes that are created for handling requests ? Is there a better way to set the capabilities of all the spawned processes ? Thanks -- Pranav Regards, Graham --
Need suggestions for adding tproxy support to mod_proxy
Hello All, I am trying to add tproxy4 (http://www.balabit.com/support/community/products/tproxy/) support to the mod_proxy to achieve transparency. It basically involves a kernel patch which allows binding of a socket to foreign address among other things. At the app layer we only need to set the setsockopt() Here is what I am thinking of doing and I would like your input as to whether this is the right approach. * Modify ap_proxy_connect_backend() to just create the connection. Basically, remove the apr_socket_connect() part. * Add a new proxy_run_pre_connection hook. * Add the tproxy specific code in a module hooked in the above hook. Just needs a setsockopt() and the client IP to bind the socket to. * And then do the connect. Is this is an overkill? Should this be done simply by modifying the ap_proxy_connect_backend() ? I would appreciate any input. Thanks -- Pranav
Re: Need suggestions for adding tproxy support to mod_proxy
On Wed, Dec 17, 2008 at 3:33 PM, Graham Leggett minf...@sharp.fm wrote: Pranav Desai wrote: I am trying to add tproxy4 (http://www.balabit.com/support/community/products/tproxy/) support to the mod_proxy to achieve transparency. It basically involves a kernel patch which allows binding of a socket to foreign address among other things. At the app layer we only need to set the setsockopt() Here is what I am thinking of doing and I would like your input as to whether this is the right approach. * Modify ap_proxy_connect_backend() to just create the connection. Basically, remove the apr_socket_connect() part. * Add a new proxy_run_pre_connection hook. * Add the tproxy specific code in a module hooked in the above hook. Just needs a setsockopt() and the client IP to bind the socket to. * And then do the connect. Is this is an overkill? Should this be done simply by modifying the ap_proxy_connect_backend() ? In theory, if it does boil down to just a single additional setsockopt(), then adding a directive to mod_proxy_connect should do the trick. Yeah, the application changes are restricted to a few lines. I believe you mean the connect_backend() and not the proxy_connect module for the CONNECT method ? It does sound like overkill to effectively come up with a whole new module. Regards, Graham --
Need suggestions for adding tproxy support to mod_proxy
Hello All, I am trying to add tproxy4 (http://www.balabit.com/support/community/products/tproxy/) support to the mod_proxy to achieve transparency. It basically involves a kernel patch which allows binding of a socket to foreign address among other things. At the app layer we only need to set the setsockopt() Here is what I am thinking of doing and I would like your input as to whether this is the right approach. * Modify ap_proxy_connect_backend() to just create the connection. Basically, remove the apr_socket_connect() part. * Add a new proxy_run_pre_connection hook. * Add the tproxy specific code in a module hooked in the above hook. Just needs a setsockopt() and the client IP to bind the socket to. * And then do the connect. Is this is an overkill? Should this be done simply by modifying the ap_proxy_connect_backend() ? I would appreciate any input. Thanks -- Pranav
