RE: [pfSense-discussion] Squid and traffic shaper
My question is this. Say you rdr your traffic to your mail server on the lan, then it tries to deliver through the router, isn't it going to just do the same thing again and keep going in a circle? I used to do http redirection for transparent cache server, but I had to put a special rule in for that cache server to allow it to go out without rdr. I did this with open bsd, not with pfsense. I may be missing something :) jason -Original Message- From: Scott Ullrich [mailto:[EMAIL PROTECTED] Sent: Thursday, November 17, 2005 1:10 PM To: discussion@pfsense.com Subject: Re: [pfSense-discussion] Squid and traffic shaper You can do that now... http://www.pfsense.com/screens/redirect_lan_to_another_mail_server.PNG Scott On 11/17/05, Dan Swartzendruber [EMAIL PROTECTED] wrote: Is it feasible to add a rdr rule to send outbound http traffic to another box on the lan? I'd do that myself
Re: [pfSense-discussion] Squid and traffic shaper
Yes, Dan pointed this out to me. We're working on a solution. On 11/18/05, Jason Brunk [EMAIL PROTECTED] wrote: My question is this. Say you rdr your traffic to your mail server on the lan, then it tries to deliver through the router, isn't it going to just do the same thing again and keep going in a circle? I used to do http redirection for transparent cache server, but I had to put a special rule in for that cache server to allow it to go out without rdr. I did this with open bsd, not with pfsense. I may be missing something :) jason -Original Message- From: Scott Ullrich [mailto:[EMAIL PROTECTED] Sent: Thursday, November 17, 2005 1:10 PM To: discussion@pfsense.com Subject: Re: [pfSense-discussion] Squid and traffic shaper You can do that now... http://www.pfsense.com/screens/redirect_lan_to_another_mail_server.PNG Scott On 11/17/05, Dan Swartzendruber [EMAIL PROTECTED] wrote: Is it feasible to add a rdr rule to send outbound http traffic to another box on the lan? I'd do that myself
Re: [pfSense-discussion] Squid and traffic shaper
This couldn't have been a more timely question. Here's a post from the author of pf that explains all you'd ever want to know about shaping. --Bill From: Daniel Hartmeier [EMAIL PROTECTED] This question pops up frequently, if this reply is too wordy, that's just so I can reference it in the future and safe typing. My apologies to the poster if this is all obvious already. ;) Rate-limiting network packets means dropping packets. It's not like a water utility pipe where you can shut the faucet incrementally and slow down the water running towards you from the water company, leaving unused water in their tanks. There are no reservoirs like that in a network (ignoring some very small buffers). If a sender is sending you packets at a rate higher than you can receive them, packets are dropped whereever there are gaps of decreasing bandwidth. And these gaps are on routers at your ISP and further upstream. Many of them will drop random packets. Some can be configured to drop based on criteria, but you don't control those criteria, because they're not your routers. Imagine you have a 1024 kbit/s downlink from your ISP to you. Assume your ISP himself has a much larger downlink himself. You're downloading a file from a web server on the Internet. Then some evil person starts sending you a flood of pings. Let's say that person has an uplink of 2048 kbit/s. Now your ISP is receiving two kinds of packets destined for you: a stream of TCP packets from the web server, and a stream of pings at 2048 kbits/. He can't possibly forward all these packets to you, since only less than half of them fit onto the link to you. So he simply forwards as many packets as he can, randomly dropping the rest. Obviously many of the TCP packets will get dropped randomly now. TCP is clever and adjust to this, the sender recognizes that there is loss between him and you, and will start to send at a lower rate. Meanwhile the flooder continues to send you pings at a happy rate of 2048 kbit/s. You'll notice how your download gets slower and slower, and you consider using rate-limiting incoming packets. You identify that the http packets are those that you prefer over the pings, and tell ALTQ to drop incoming pings exceeding, say, a harmless rate of 64 kbit/s and reserve the rest for the more important http packets. Fine, it could do that. But it wouldn't change anything, because the congestion is upstream of your ALTQ box. You can drop as many packets as you like after you received them, that doesn't free up any bandwidth on your downlink. The downlink will continue to carry mostly ping packets, because you dropping packets has no influence on what happens at either sender, at your ISP's router dropping random packets, or on your downlink. Just like the rate of water you can draw from your water line isn't influenced by what you do with the water that has already come out of it. Rate-limiting is not like a faucet. This is the reason you often get the answer It just doesn't make sense to rate-limit incoming packets, and I guess that's the reason why ALTQ simply doesn't add queues for incoming packets, but only outgoing ones. Now, if we forget about the DoS case, and assume you have only flow-controlled TCP connections with cooperative peers, things are a little different. If you receive two streams of TCP packets, and you start dropping packets of one stream (after you have received them and they have taken up bandwidth on your downlink), the corresponding peer will detect that loss and helpfully slow down sending, freeing up bandwidth on your downlink for the other peer. In fact, if you tell your ALTQ box to limit one stream to, say, 128 kbit/s, and drop all excess packets, that peer will (eventually, by trial and error) come to the conclusion that sending you packets at a rate of about 128 kbit/s is the optimal thing to do. But it's important to realize that you're not really enforcing this limit, but it's the peer that kindly reacts in that way. If you want to do this with ALTQ, you can do so by limiting outgoing packets on the other interface, assuming the box is forwarding all packets between two interfaces. If a browser (on a separate local box) is downloading a file from an external web server _through_ the ALTQ box, you rate-limit packets going out through the internal interface. Every packet coming in on the external interface obviously goes out through the internal interface, hence rate-limiting outgoing packets on the internal interface has the same effect as rate-limiting incoming packets on the external interface. This does not work if the client is on the ALTQ box itself, obviously (there is no other interface to rate-limit on). In this case you're facing a limitation of ALTQ itself. You might have to move ALTQ onto an additional intermediate box, just so you do have a second interface. I don't think there are any plans to introduce incoming queues in ALTQ. On 11/17/05, Kieron Mohindra [EMAIL PROTECTED] wrote: Hi
Re: [pfSense-discussion] Squid and traffic shaper
Only hack I can think of is to lie and tell the traffic shaper that your inbound pipe is 10mb (or whatever.)
RE: [pfSense-discussion] Squid and traffic shaper
Ah, that's sharpened up my understanding of traffic shaping quite a bit. So, the bottom line is that to really make the most of squid, you should be running it on another box? I could do the hack of upping the download limit, but this would remove any benefits of traffic shaping in that direction. Thanks for the very comprehensive reply! Kieron -Original Message- From: Bill Marquette [mailto:[EMAIL PROTECTED] Sent: 17 November 2005 15:10 To: discussion@pfsense.com Subject: Re: [pfSense-discussion] Squid and traffic shaper This couldn't have been a more timely question. Here's a post from the author of pf that explains all you'd ever want to know about shaping. --Bill From: Daniel Hartmeier [EMAIL PROTECTED] This question pops up frequently, if this reply is too wordy, that's just so I can reference it in the future and safe typing. My apologies to the poster if this is all obvious already. ;) Rate-limiting network packets means dropping packets. It's not like a water utility pipe where you can shut the faucet incrementally and slow down the water running towards you from the water company, leaving unused water in their tanks. There are no reservoirs like that in a network (ignoring some very small buffers). If a sender is sending you packets at a rate higher than you can receive them, packets are dropped whereever there are gaps of decreasing bandwidth. And these gaps are on routers at your ISP and further upstream. Many of them will drop random packets. Some can be configured to drop based on criteria, but you don't control those criteria, because they're not your routers. Imagine you have a 1024 kbit/s downlink from your ISP to you. Assume your ISP himself has a much larger downlink himself. You're downloading a file from a web server on the Internet. Then some evil person starts sending you a flood of pings. Let's say that person has an uplink of 2048 kbit/s. Now your ISP is receiving two kinds of packets destined for you: a stream of TCP packets from the web server, and a stream of pings at 2048 kbits/. He can't possibly forward all these packets to you, since only less than half of them fit onto the link to you. So he simply forwards as many packets as he can, randomly dropping the rest. Obviously many of the TCP packets will get dropped randomly now. TCP is clever and adjust to this, the sender recognizes that there is loss between him and you, and will start to send at a lower rate. Meanwhile the flooder continues to send you pings at a happy rate of 2048 kbit/s. You'll notice how your download gets slower and slower, and you consider using rate-limiting incoming packets. You identify that the http packets are those that you prefer over the pings, and tell ALTQ to drop incoming pings exceeding, say, a harmless rate of 64 kbit/s and reserve the rest for the more important http packets. Fine, it could do that. But it wouldn't change anything, because the congestion is upstream of your ALTQ box. You can drop as many packets as you like after you received them, that doesn't free up any bandwidth on your downlink. The downlink will continue to carry mostly ping packets, because you dropping packets has no influence on what happens at either sender, at your ISP's router dropping random packets, or on your downlink. Just like the rate of water you can draw from your water line isn't influenced by what you do with the water that has already come out of it. Rate- limiting is not like a faucet. This is the reason you often get the answer It just doesn't make sense to rate-limit incoming packets, and I guess that's the reason why ALTQ simply doesn't add queues for incoming packets, but only outgoing ones. Now, if we forget about the DoS case, and assume you have only flow-controlled TCP connections with cooperative peers, things are a little different. If you receive two streams of TCP packets, and you start dropping packets of one stream (after you have received them and they have taken up bandwidth on your downlink), the corresponding peer will detect that loss and helpfully slow down sending, freeing up bandwidth on your downlink for the other peer. In fact, if you tell your ALTQ box to limit one stream to, say, 128 kbit/s, and drop all excess packets, that peer will (eventually, by trial and error) come to the conclusion that sending you packets at a rate of about 128 kbit/s is the optimal thing to do. But it's important to realize that you're not really enforcing this limit, but it's the peer that kindly reacts in that way. If you want to do this with ALTQ, you can do so by limiting outgoing packets on the other interface, assuming the box is forwarding all packets between two interfaces. If a browser (on a separate local box) is downloading a file from an external web server _through_ the ALTQ box, you rate-limit packets going out through the internal interface. Every packet coming
Re: [pfSense-discussion] Squid and traffic shaper
The hack almost works, but I've had issues with it myself and I'm still working on tracking down the bug that's causing the traffic to end up in the wrong queue. --Bill On 11/17/05, Kieron Mohindra [EMAIL PROTECTED] wrote: Ah, that's sharpened up my understanding of traffic shaping quite a bit. So, the bottom line is that to really make the most of squid, you should be running it on another box? I could do the hack of upping the download limit, but this would remove any benefits of traffic shaping in that direction. Thanks for the very comprehensive reply! Kieron -Original Message- From: Bill Marquette [mailto:[EMAIL PROTECTED] Sent: 17 November 2005 15:10 To: discussion@pfsense.com Subject: Re: [pfSense-discussion] Squid and traffic shaper This couldn't have been a more timely question. Here's a post from the author of pf that explains all you'd ever want to know about shaping. --Bill From: Daniel Hartmeier [EMAIL PROTECTED] This question pops up frequently, if this reply is too wordy, that's just so I can reference it in the future and safe typing. My apologies to the poster if this is all obvious already. ;) Rate-limiting network packets means dropping packets. It's not like a water utility pipe where you can shut the faucet incrementally and slow down the water running towards you from the water company, leaving unused water in their tanks. There are no reservoirs like that in a network (ignoring some very small buffers). If a sender is sending you packets at a rate higher than you can receive them, packets are dropped whereever there are gaps of decreasing bandwidth. And these gaps are on routers at your ISP and further upstream. Many of them will drop random packets. Some can be configured to drop based on criteria, but you don't control those criteria, because they're not your routers. Imagine you have a 1024 kbit/s downlink from your ISP to you. Assume your ISP himself has a much larger downlink himself. You're downloading a file from a web server on the Internet. Then some evil person starts sending you a flood of pings. Let's say that person has an uplink of 2048 kbit/s. Now your ISP is receiving two kinds of packets destined for you: a stream of TCP packets from the web server, and a stream of pings at 2048 kbits/. He can't possibly forward all these packets to you, since only less than half of them fit onto the link to you. So he simply forwards as many packets as he can, randomly dropping the rest. Obviously many of the TCP packets will get dropped randomly now. TCP is clever and adjust to this, the sender recognizes that there is loss between him and you, and will start to send at a lower rate. Meanwhile the flooder continues to send you pings at a happy rate of 2048 kbit/s. You'll notice how your download gets slower and slower, and you consider using rate-limiting incoming packets. You identify that the http packets are those that you prefer over the pings, and tell ALTQ to drop incoming pings exceeding, say, a harmless rate of 64 kbit/s and reserve the rest for the more important http packets. Fine, it could do that. But it wouldn't change anything, because the congestion is upstream of your ALTQ box. You can drop as many packets as you like after you received them, that doesn't free up any bandwidth on your downlink. The downlink will continue to carry mostly ping packets, because you dropping packets has no influence on what happens at either sender, at your ISP's router dropping random packets, or on your downlink. Just like the rate of water you can draw from your water line isn't influenced by what you do with the water that has already come out of it. Rate- limiting is not like a faucet. This is the reason you often get the answer It just doesn't make sense to rate-limit incoming packets, and I guess that's the reason why ALTQ simply doesn't add queues for incoming packets, but only outgoing ones. Now, if we forget about the DoS case, and assume you have only flow-controlled TCP connections with cooperative peers, things are a little different. If you receive two streams of TCP packets, and you start dropping packets of one stream (after you have received them and they have taken up bandwidth on your downlink), the corresponding peer will detect that loss and helpfully slow down sending, freeing up bandwidth on your downlink for the other peer. In fact, if you tell your ALTQ box to limit one stream to, say, 128 kbit/s, and drop all excess packets, that peer will (eventually, by trial and error) come to the conclusion that sending you packets at a rate of about 128 kbit/s is the optimal thing to do. But it's important to realize that you're not really enforcing this limit, but it's the peer that kindly reacts in that way. If you want to do this with ALTQ, you can do so by limiting
RE: [pfSense-discussion] Squid and traffic shaper
Is it feasible to add a rdr rule to send outbound http traffic to another box on the lan? I'd do that myself
Re: [pfSense-discussion] Squid and traffic shaper
You can do that now... http://www.pfsense.com/screens/redirect_lan_to_another_mail_server.PNG Scott On 11/17/05, Dan Swartzendruber [EMAIL PROTECTED] wrote: Is it feasible to add a rdr rule to send outbound http traffic to another box on the lan? I'd do that myself
