Unsubscribe is here: http://lists.pfsense.org/mailman/listinfo/list
On 3/19/14, Edouard De Keyser <edou...@ipfix.be> wrote: > Please stop your mail. > Thank you > > Envoyé de mon SkyTel > >> Le 19 mars 2014 à 20:29, Chris Buechler <c...@pfsense.com> a écrit : >> >> It sounds like you don't have state sync enabled on the secondary, it >> won't accept the primary's states without that. >> >> Depending on how much load you're generating with the crawlers, you >> could be hitting the limits of the ALIX in new connections per sec. >> I've seen with one customer where they were blasting out 10K+ emails >> (and 10K+ SMTP connections) in less than a second, which put adequate >> load on their ALIX pair that it failed over CARP because the primary >> was under too much load to send its advertisements. >> >> Though the modem theory is just as plausible, especially if the modem >> is doing any kind of NAT or filtering. If you're not hitting it so >> hard you're failing over CARP, that points to it being something other >> than the firewall. Packet capture on WAN filtered on port 53 would be >> more telling. If you see DNS queries leaving there that get no reply >> back, it's not the firewall. >> >> >>> On Wed, Mar 19, 2014 at 9:50 AM, David Noel <david.i.n...@gmail.com> >>> wrote: >>> Well, it may not be the ALIX boards after all. I connected the servers >>> directly to the modem, ran the crawlers, and I'm still getting >>> UnknownHostException's. I'm guessing my modem's to blame... I'll have >>> to upgrade it and find out. >>> >>>> On 3/18/14, David Noel <david.i.n...@gmail.com> wrote: >>>> Well, I bumped Maximum State Table from the default of 23,000 to >>>> 75,000, and now it's throwing fewer UnknownHostException's. But >>>> they're still being thrown. My resource utilization is getting pretty >>>> high though. I don't think these ALIX boards can handle much more of a >>>> load, and I still have 2 more servers I need to scale these crawlers >>>> out to. I do see there's a "Firewall Adaptive Timeouts" setting in the >>>> web configurator.. this seems like it might be useful. Can anyone >>>> recommend any settings I should try to free up some system resources? >>>> I'm not clear on the consequences of purging pf state entries and >>>> whether that's something I'd want to do though. >>>> >>>> The state table on my primary router (alix1) is at roughly 50% >>>> utilization, or 40,000 states. The state table on my secondary router >>>> (alix2) is at 0%, roughly 250 states. This seems odd. Is this to be >>>> expected under CARP? Why is the load not distributed evenly? >>>> >>>> Memory usage on my primary router (alix1) is hovering around 55% (of >>>> 235MB). On my backup (alix2) it's pushing 85-90%. Does this make sense >>>> to anyone? Top output looks roughly the same... and now alix2 has gone >>>> down. 95% packet loss. Web Configurator unresponsive. ... It's back up >>>> but throwing "500 - Internal Server Error"s periodically. I've ssh'd >>>> in to alix2 and am looking at top output.. tcpdump seems to be running >>>> for pflog purposes.. and it's hogging quite a bit of CPU. Is this >>>> necessary? Can I disable it somehow? >>>> >>>> -David >>>> >>>>> On 3/18/14, David Noel <david.i.n...@gmail.com> wrote: >>>>> I've encountered a strange issue while scaling a Java project that I'm >>>>> not quite sure how to resolve. Any thoughts would be appreciated. >>>>> >>>>> The code is a crawler that uses HTMLUnit to crawl a bunch of pages >>>>> concurrently. It uses HTMLUnits getPage method to do the crawling. I'm >>>>> running 100 threads per instance. When I have 1 instance up and >>>>> running on 1 machine everything is fine. When I scale it to a second >>>>> machine though I start having trouble. Calls to getPage keep throwing >>>>> UnknownHostException's (DNS resolution error). With 2 servers running, >>>>> roughly 1 out of every 20 calls to getPage throw this exception. For >>>>> some reason it's unable to resolve domain names.. and it's not just >>>>> the crawlers, my entire network starts to bug on DNS queries. On >>>>> different systems on the same network I get 'unable to resolve host' >>>>> errors in my web browser periodically when loading URL's. Usually when >>>>> I retry it goes through, but it keeps happening sporadically as long >>>>> as the crawlers are running. >>>>> >>>>> So many things could be going wrong here. Thinking maybe it was my >>>>> provider throttling DNS queries I've tried changing DNS servers, but >>>>> that's done nothing. Thinking it might be a bandwidth issue I checked >>>>> systat, but the cumulative load is well under what my line can handle. >>>>> What else could be causing this? My network is pretty simple: Provider >>>>> <--> modem <--> 2 ALIX boards running pfSense <--> Servers and >>>>> workstations. The servers are running FreeBSD, and the workstations >>>>> run FreeBSD, Windows, and OSX. >>>>> >>>>> Has anyone encountered this before? Does anyone have any thoughts on >>>>> what might be causing it? >>>>> >>>>> My only other thought is that maybe pfSense is doing something strange >>>>> so if I can't come up with any better ideas I'll try plugging the >>>>> servers directly into the modem. I'd rather have them behind the >>>>> routers though, so this would be a less-than-ideal solution. >>>>> >>>>> UPDATE: Ok, so it seems to be a pfSense issue. I launched the crawlers >>>>> on 2 servers as before and waited for UnknownHostException's to be >>>>> thrown. I then took a spare laptop and connected it directly into my >>>>> modem, bypassing my 2 pfSense routers. All DNS queries have gone >>>>> through without a hitch, so something strange is going on with >>>>> pfSense. Can anyone think of what might be causing this? I'm guessing >>>>> there's some tunable that needs to be tweaked, but I'm not sure where >>>>> to start. I also might have configured pfSense incorrectly, but I >>>>> think that's less likely to be the case than some default tunable >>>>> being set too low because at low volumes all DNS queries go through >>>>> just fine. If it were a configuration error it seems more likely that >>>>> no DNS queries would be going through. If it's relevant, with 200 >>>>> active threads I'm probably querying DNS a minimum of 10 times per >>>>> second. >>>>> >>>>> Can anyone think of anything I might have done wrong setting up >>>>> pfSense? Does anyone know of any tunables that might causing this >>>>> error? >>> _______________________________________________ >>> List mailing list >>> List@lists.pfsense.org >>> https://lists.pfsense.org/mailman/listinfo/list >> _______________________________________________ >> List mailing list >> List@lists.pfsense.org >> https://lists.pfsense.org/mailman/listinfo/list > _______________________________________________ List mailing list List@lists.pfsense.org https://lists.pfsense.org/mailman/listinfo/list
