To me that looks like the remote servers issue, you could be hammering it, put a sleep command in to let it recover a little.
Jonathan On 28 July 2010 10:41, Robbie Shepherd <[email protected]> wrote: > Its a rails app, deployed with passenger and apache, but I get the same > issues when running it locally on os x as well. > > Mark - that's a good idea, will give that a go, see if I can bypass hpricot > for getting the response. > > I'll try and get a better stack trace, will reply in a bit. Thanks for the > suggestions so far. > > Robbie > > > On Wed, Jul 28, 2010 at 10:37 AM, Lachie <[email protected]> wrote: >> >> Its the app at the other end with the problem "500 Internal Server Error" >> >> Is that within your control too? Can you give us more details on that >> app, and how its deployed? >> >> e.g. is it a rails app deployed with passenger behind apache? >> >> :lachie >> http://plus2.com.au >> http://smartbomb.com.au >> http://www.flickr.com/photos/lachie/ >> >> >> >> On Wed, Jul 28, 2010 at 10:29 AM, Robbie Shepherd >> <[email protected]> wrote: >> > Simon, yes they are. >> > >> > Julio, I'm getting the same error on both os x and my server running >> > Ubuntu >> > >> > Robbie >> > >> > On Wed, Jul 28, 2010 at 10:28 AM, Simon Russell <[email protected]> >> > wrote: >> >> >> >> Are they all coming from the same server? >> >> >> >> On Wed, Jul 28, 2010 at 10:25, Mark Wotton <[email protected]> wrote: >> >> > On three URLs? Surely not... >> >> > >> >> > mark >> >> > >> >> > On Wed, Jul 28, 2010 at 10:24 AM, Julio Cesar Ody >> >> > <[email protected]> >> >> > wrote: >> >> >> I can't really tell from that stacktrace if the problem is this, but >> >> >> you may be hitting the maximum number of open files (and sockets). >> >> >> >> >> >> Are you on OSX? If so, you can expand that limit with sysctl. Try: >> >> >> >> >> >> sudo sysctl -w kern.maxfiles=36000 >> >> >> >> >> >> Sets to a bit over twice as much the default. Also, check >> >> >> kern.maxfilesperproc. And ulimit. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd >> >> >> <[email protected]> wrote: >> >> >>> >> >> >>> Does anyone know if there's a maximum number of open connections >> >> >>> allowed when using hpricot/net::http? >> >> >>> >> >> >>> I have a bunch of xml feeds I need to read via hpricot and write to >> >> >>> disk. Whenever I hit it too hard/fast, or when I try to open >> >> >>> several files >> >> >>> in hpricot in a loop, my app dies. >> >> >>> >> >> >>> If I do something like... >> >> >>> >> >> >>> arr = [url1, url2, url2] >> >> >>> arr.each do|d| >> >> >>> doc = Hpricot(open(d)) >> >> >>> doc.search(blah...) >> >> >>> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write >> >> >>> doc } >> >> >>> end >> >> >>> >> >> >>> I get: >> >> >>> >> >> >>> OpenURI::HTTPError: 500 Internal Server Error from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in >> >> >>> `open_http' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in >> >> >>> `buffer_open' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in >> >> >>> `open_loop' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >> >> >>> `catch' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >> >> >>> `open_loop' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in >> >> >>> `open_uri' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in >> >> >>> `open' >> >> >>> from >> >> >>> >> >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in >> >> >>> `open' >> >> >>> from (irb):11 >> >> >>> from (irb):11:in `each' >> >> >>> from (irb):11 >> >> >>> >> >> >>> Am I missing something here, or is there a better way of doing >> >> >>> this? >> >> >>> I'm on Ruby 1.8.7 and hpricot 0.8.2 >> >> >>> >> >> >>> Thanks >> >> >>> Robbie >> >> >>> >> >> >>> -- >> >> >>> You received this message because you are subscribed to the Google >> >> >>> Groups "Ruby or Rails Oceania" group. >> >> >>> To post to this group, send email to >> >> >>> [email protected]. >> >> >>> To unsubscribe from this group, send email to >> >> >>> [email protected]. >> >> >>> For more options, visit this group at >> >> >>> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> http://crazyhollywood.org >> >> >> >> >> >> -- >> >> >> You received this message because you are subscribed to the Google >> >> >> Groups "Ruby or Rails Oceania" group. >> >> >> To post to this group, send email to [email protected]. >> >> >> To unsubscribe from this group, send email to >> >> >> [email protected]. >> >> >> For more options, visit this group at >> >> >> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > A UNIX signature isn't a return address, it's the ASCII equivalent of >> >> > a >> >> > black velvet clown painting. It's a rectangle of carets surrounding a >> >> > quote from a literary giant of weeniedom like Heinlein or Dr. Who. >> >> > -- Chris Maeda >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "Ruby or Rails Oceania" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group at >> >> > http://groups.google.com/group/rails-oceania?hl=en. >> >> > >> >> > >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "Ruby or Rails Oceania" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Ruby or Rails Oceania" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/rails-oceania?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby or Rails Oceania" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/rails-oceania?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
