Re: [pfSense] DNS resolution issues under heavy load
Unsubscribe is here: http://lists.pfsense.org/mailman/listinfo/list On 3/19/14, Edouard De Keyser edou...@ipfix.be wrote: Please stop your mail. Thank you Envoyé de mon SkyTel Le 19 mars 2014 à 20:29, Chris Buechler c...@pfsense.com a écrit : It sounds like you don't have state sync enabled on the secondary, it won't accept the primary's states without that. Depending on how much load you're generating with the crawlers, you could be hitting the limits of the ALIX in new connections per sec. I've seen with one customer where they were blasting out 10K+ emails (and 10K+ SMTP connections) in less than a second, which put adequate load on their ALIX pair that it failed over CARP because the primary was under too much load to send its advertisements. Though the modem theory is just as plausible, especially if the modem is doing any kind of NAT or filtering. If you're not hitting it so hard you're failing over CARP, that points to it being something other than the firewall. Packet capture on WAN filtered on port 53 would be more telling. If you see DNS queries leaving there that get no reply back, it's not the firewall. On Wed, Mar 19, 2014 at 9:50 AM, David Noel david.i.n...@gmail.com wrote: Well, it may not be the ALIX boards after all. I connected the servers directly to the modem, ran the crawlers, and I'm still getting UnknownHostException's. I'm guessing my modem's to blame... I'll have to upgrade it and find out. On 3/18/14, David Noel david.i.n...@gmail.com wrote: Well, I bumped Maximum State Table from the default of 23,000 to 75,000, and now it's throwing fewer UnknownHostException's. But they're still being thrown. My resource utilization is getting pretty high though. I don't think these ALIX boards can handle much more of a load, and I still have 2 more servers I need to scale these crawlers out to. I do see there's a Firewall Adaptive Timeouts setting in the web configurator.. this seems like it might be useful. Can anyone recommend any settings I should try to free up some system resources? I'm not clear on the consequences of purging pf state entries and whether that's something I'd want to do though. The state table on my primary router (alix1) is at roughly 50% utilization, or 40,000 states. The state table on my secondary router (alix2) is at 0%, roughly 250 states. This seems odd. Is this to be expected under CARP? Why is the load not distributed evenly? Memory usage on my primary router (alix1) is hovering around 55% (of 235MB). On my backup (alix2) it's pushing 85-90%. Does this make sense to anyone? Top output looks roughly the same... and now alix2 has gone down. 95% packet loss. Web Configurator unresponsive. ... It's back up but throwing 500 - Internal Server Errors periodically. I've ssh'd in to alix2 and am looking at top output.. tcpdump seems to be running for pflog purposes.. and it's hogging quite a bit of CPU. Is this necessary? Can I disable it somehow? -David On 3/18/14, David Noel david.i.n...@gmail.com wrote: I've encountered a strange issue while scaling a Java project that I'm not quite sure how to resolve. Any thoughts would be appreciated. The code is a crawler that uses HTMLUnit to crawl a bunch of pages concurrently. It uses HTMLUnits getPage method to do the crawling. I'm running 100 threads per instance. When I have 1 instance up and running on 1 machine everything is fine. When I scale it to a second machine though I start having trouble. Calls to getPage keep throwing UnknownHostException's (DNS resolution error). With 2 servers running, roughly 1 out of every 20 calls to getPage throw this exception. For some reason it's unable to resolve domain names.. and it's not just the crawlers, my entire network starts to bug on DNS queries. On different systems on the same network I get 'unable to resolve host' errors in my web browser periodically when loading URL's. Usually when I retry it goes through, but it keeps happening sporadically as long as the crawlers are running. So many things could be going wrong here. Thinking maybe it was my provider throttling DNS queries I've tried changing DNS servers, but that's done nothing. Thinking it might be a bandwidth issue I checked systat, but the cumulative load is well under what my line can handle. What else could be causing this? My network is pretty simple: Provider -- modem -- 2 ALIX boards running pfSense -- Servers and workstations. The servers are running FreeBSD, and the workstations run FreeBSD, Windows, and OSX. Has anyone encountered this before? Does anyone have any thoughts on what might be causing it? My only other thought is that maybe pfSense is doing something strange so if I can't come up with any better ideas I'll try plugging the servers directly into the modem. I'd rather have them behind the routers though, so this would be a less-than-ideal solution. UPDATE: Ok, so it seems to be a pfSense issue. I
Re: [pfSense] Odd symptoms from embedded 2.1-RELEASE
On Thu, Mar 20, 2014 at 10:12 AM, Ryan Coleman ryanjc...@me.com wrote: So I’m going to try and fix it if there’s someone that is willing to help me out today.. this just blows my mind - it’s like it loses the firewall configuration and then falls to a default. None of the VLANs are passing data. Have you ruled out power supply issues? db ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] Odd symptoms from embedded 2.1-RELEASE
Funny you mention that… that is the ONLY piece I haven’t swapped out but I have a good one with me. I brought the one that came with the NetGate ALIX board with me today and just swapped that out. The one I had in place was a 12V 1A adapter - more than enough to run the board. The 15V that came with it is now in play and it is booting back up. On Mar 20, 2014, at 11:16 AM, David Burgess apt@gmail.com wrote: On Thu, Mar 20, 2014 at 10:12 AM, Ryan Coleman ryanjc...@me.com wrote: So I’m going to try and fix it if there’s someone that is willing to help me out today.. this just blows my mind - it’s like it loses the firewall configuration and then falls to a default. None of the VLANs are passing data. Have you ruled out power supply issues? db ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] Odd symptoms from embedded 2.1-RELEASE
Filtered to show just the network I’m on… 50 matched log entries.Max(50) Act TimeIf Source Destination Proto Mar 20 11:21:59 BACKOFHOUSE 10.20.1.12:3738631.13.74.128:443 TCP:FA Mar 20 11:22:59 BACKOFHOUSE 10.20.1.12:39703 173.252.102.16:443TCP:FPA Mar 20 11:23:36 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:PA Mar 20 11:23:36 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:37 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:38 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:40 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:41 BACKOFHOUSE 10.20.1.12:58990 173.252.102.16:443TCP:FPA Mar 20 11:23:44 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:52 BACKOFHOUSE 10.20.1.12:3738631.13.74.128:443 TCP:FA Mar 20 11:24:00 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:24:41 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:41 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:42 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:42 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:43 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:45 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:47 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:51 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:24:53 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:24:55 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:25:12 BACKOFHOUSE 10.20.1.12:39703 173.252.102.16:443TCP:FPA Mar 20 11:26:24 BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:26:53 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:53 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:54 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:55 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:08 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:17 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:30 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:53 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:28:43 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:30:17 BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:32:10 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:PA Mar 20 11:32:10 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:PA Mar 20 11:32:10 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:PA Mar 20 11:32:10 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FA Mar 20 11:32:11 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:32:11 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:32:18 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:32:39 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:32:43 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:33:00 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:33:53 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Mar 20 11:34:03 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:34:07 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:34:11 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:34:15 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:FA Mar 20 11:34:20 BACKOFHOUSE 10.20.1.18:5150763.156.193.136:80 TCP:RA Mar 20 11:35:01 BACKOFHOUSE 10.20.1.12:34519173.194.46.52:443 TCP:FPA Design: Century Link DSL Modem NetGate 2D13 Netgear GS110TPNetgear GS110TP POS systems - non-routed/controlled
Re: [pfSense] Odd symptoms from embedded 2.1-RELEASE
I put the device that was working from home last night on the network with the configuration unchanged and it’s working again. Is this a situation I need to consider using CARP for? On Mar 20, 2014, at 11:44 AM, Ryan Coleman ryanjc...@me.com wrote: Filtered to show just the network I’m on… 50 matched log entries.Max(50) Act TimeIf Source Destination Proto Mar 20 11:21:59BACKOFHOUSE 10.20.1.12:37386 31.13.74.128:443 TCP:FA Mar 20 11:22:59BACKOFHOUSE 10.20.1.12:39703 173.252.102.16:443TCP:FPA Mar 20 11:23:36BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:PA Mar 20 11:23:36BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:37BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:38BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:40BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:41BACKOFHOUSE 10.20.1.12:58990 173.252.102.16:443TCP:FPA Mar 20 11:23:44BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:23:52BACKOFHOUSE 10.20.1.12:37386 31.13.74.128:443 TCP:FA Mar 20 11:24:00BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:24:41BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:41BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:42BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:42BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:43BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:45BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:47BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:51BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:24:53BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:24:55BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar 20 11:25:12BACKOFHOUSE 10.20.1.12:39703 173.252.102.16:443TCP:FPA Mar 20 11:26:24BACKOFHOUSE 10.20.1.12:37384 74.125.225.142:443TCP:FPA Mar 20 11:26:53BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:53BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:54BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:26:55BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:08BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:17BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:30BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:PA Mar 20 11:27:53BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:28:43BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:30:17BACKOFHOUSE 10.20.1.12:54459 74.125.142.188:5228 TCP:FPA Mar 20 11:32:10BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:PA Mar 20 11:32:10BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:PA Mar 20 11:32:10BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:PA Mar 20 11:32:10BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FA Mar 20 11:32:11BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:32:11BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:32:18BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:32:39BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:32:43BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:33:00BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:33:53BACKOFHOUSE 10.20.1.12:34519 173.194.46.52:443 TCP:FPA Mar 20 11:34:03BACKOFHOUSE 10.20.1.18:51507 63.156.193.136:80 TCP:FA Mar
Re: [pfSense] Proxy filter
On 20/3/14 7:14 pm, A Mohan Rao wrote: I m using squid squid guard and light squid for user access websites reporting with live but there is pfsense not read or show ftp server access logs. I also try as pfsense firewall client and to to any other ftp sites then download files but in proxy filter tab not show my ip logs. Pls guide where i watch ftp access logs I might be misunderstanding the issue here, but Squid is an HTTP proxy - it's not going to do anything to filter or proxy FTP traffic at all. Kind regards, Chris -- This email is made from 100% recycled electrons ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] Proxy filter
On 20/3/14 7:19 pm, A Mohan Rao wrote: Ok thanks but if i need how i maintain ftp traffic logs. Not really relevant to the question, I appreciate, but I can't think of a good reason why you'd want to do that, unless of course you're running the FTP server, in which case your FTP server should have that ability in its settings. You might be able to do something using a span port on a switch and some clever logging rules, but that's outside my scope. Perhaps there's another pfSense package that'll do what you want? Kind regards, Chris -- This email is made from 100% recycled electrons ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] Proxy filter
On 20/3/14 7:22 pm, A Mohan Rao wrote: Also i struggling to block https social networking sites like facebook etc from last 1to 1.5 years. I used for block that domain through DNS FORWARDER. But when user use open dns its working pls any idea its very helpful for me. You might find it easier to block OpenDNS than blocking the site itself. If you were to add a LAN rule that blocks traffic on destination port 53 to anything apart from the pfSense interface IP, you'll probably be able to block most external DNS services. That won't, of course, prevent users from tunnelling their traffic through VPN services and the like. Though as I said in my earlier email, I'm not sure I understand why you want to block things so forcefully. User education (e.g. explaining to colleagues why it's inappropriate to access Facebook during work hours) nearly always works better than technical blocks. Kind regards, Chris -- This email is made from 100% recycled electrons ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] Proxy filter
Ok, Actually i have 600 internet users and i have 22 Mbps leased line. I m not gave any users to full permission but some users are go to out of the way with lots of free proxy sites download videos or movies thats why i need to watch that user https and ftp traffic. Regards Mohan On Mar 21, 2014 12:59 AM, Chris Bagnall pfse...@lists.minotaur.cc wrote: On 20/3/14 7:19 pm, A Mohan Rao wrote: Ok thanks but if i need how i maintain ftp traffic logs. Not really relevant to the question, I appreciate, but I can't think of a good reason why you'd want to do that, unless of course you're running the FTP server, in which case your FTP server should have that ability in its settings. You might be able to do something using a span port on a switch and some clever logging rules, but that's outside my scope. Perhaps there's another pfSense package that'll do what you want? Kind regards, Chris -- This email is made from 100% recycled electrons ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
Re: [pfSense] pfsense openvpn Road Warrior
Hi Mohan, You can see the routes for your internal network by typing command route print in windows cmd, you can see if any routes configured by you for openvpn are being pushed to the clients, and yes i maintain a pfsense based openvpn setup. On Thu, Mar 20, 2014 at 12:39 AM, A Mohan Rao mohanra...@gmail.com wrote: Hello RAJAN JI, No, can u pls guide me where i see or configure internal routes. Have u successfully configured Road Warrior open vpn or it is running any where. If u want any other info pls tell me i will provide u immidiately. Like gui or any viewer. Thanks Mohan On Mar 19, 2014 4:01 PM, rajan agarwal rajanagarwa...@gmail.com wrote: Hi Mohan, Can you see if the internal routes of your network are pushed to the client using route print command in cmd? On Wed, Mar 19, 2014 at 3:55 PM, A Mohan Rao mohanra...@gmail.comwrote: yes client is properly connected from windows computer system. also show that client name in tab open vpn status page . but not able to communication between. thanks Mohan On Wed, Mar 19, 2014 at 3:37 PM, rajan agarwal rajanagarwa...@gmail.com wrote: Hi Mohan, Are you using windows for connections? Can you see if routes were added when client connected to VPN or not. On Wed, Mar 19, 2014 at 2:54 PM, A Mohan Rao mohanra...@gmail.comwrote: Hello Team, Hello, i have configured openvpn road warrior also client is properly connected from outside internet network. but not able to access server end network and servers's. can anybody give any help where is do any wrong steps. Thanks Mohan ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list ___ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
