I can't really tell from that stacktrace if the problem is this, but you may be hitting the maximum number of open files (and sockets).
Are you on OSX? If so, you can expand that limit with sysctl. Try: sudo sysctl -w kern.maxfiles=36000 Sets to a bit over twice as much the default. Also, check kern.maxfilesperproc. And ulimit. On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd <[email protected]> wrote: > > Does anyone know if there's a maximum number of open connections allowed when > using hpricot/net::http? > > I have a bunch of xml feeds I need to read via hpricot and write to disk. > Whenever I hit it too hard/fast, or when I try to open several files in > hpricot in a loop, my app dies. > > If I do something like... > > arr = [url1, url2, url2] > arr.each do|d| > doc = Hpricot(open(d)) > doc.search(blah...) > File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write doc } > end > > I get: > > OpenURI::HTTPError: 500 Internal Server Error from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in > `open_http' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in > `buffer_open' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in > `open_loop' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > `catch' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in > `open_loop' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in > `open_uri' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in > `open' > from > /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in > `open' > from (irb):11 > from (irb):11:in `each' > from (irb):11 > > Am I missing something here, or is there a better way of doing this? I'm on > Ruby 1.8.7 and hpricot 0.8.2 > > Thanks > Robbie > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. -- http://crazyhollywood.org -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
