Are they all coming from the same server? On Wed, Jul 28, 2010 at 10:25, Mark Wotton <[email protected]> wrote: > On three URLs? Surely not... > > mark > > On Wed, Jul 28, 2010 at 10:24 AM, Julio Cesar Ody <[email protected]> wrote: >> I can't really tell from that stacktrace if the problem is this, but >> you may be hitting the maximum number of open files (and sockets). >> >> Are you on OSX? If so, you can expand that limit with sysctl. Try: >> >> sudo sysctl -w kern.maxfiles=36000 >> >> Sets to a bit over twice as much the default. Also, check >> kern.maxfilesperproc. And ulimit. >> >> >> >> >> On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd >> <[email protected]> wrote: >>> >>> Does anyone know if there's a maximum number of open connections allowed >>> when using hpricot/net::http? >>> >>> I have a bunch of xml feeds I need to read via hpricot and write to disk. >>> Whenever I hit it too hard/fast, or when I try to open several files in >>> hpricot in a loop, my app dies. >>> >>> If I do something like... >>> >>> arr = [url1, url2, url2] >>> arr.each do|d| >>> doc = Hpricot(open(d)) >>> doc.search(blah...) >>> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write doc } >>> end >>> >>> I get: >>> >>> OpenURI::HTTPError: 500 Internal Server Error from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in >>> `open_http' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in >>> `buffer_open' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in >>> `open_loop' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >>> `catch' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >>> `open_loop' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in >>> `open_uri' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in >>> `open' >>> from >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in >>> `open' >>> from (irb):11 >>> from (irb):11:in `each' >>> from (irb):11 >>> >>> Am I missing something here, or is there a better way of doing this? I'm on >>> Ruby 1.8.7 and hpricot 0.8.2 >>> >>> Thanks >>> Robbie >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Ruby or Rails Oceania" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> >> -- >> http://crazyhollywood.org >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby or Rails Oceania" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/rails-oceania?hl=en. >> >> > > > > -- > A UNIX signature isn't a return address, it's the ASCII equivalent of a > black velvet clown painting. It's a rectangle of carets surrounding a > quote from a literary giant of weeniedom like Heinlein or Dr. Who. > -- Chris Maeda > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > >
-- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
