can you just dump the response instead of passing it through Hpricot? it sounds like hpricot is pretty tangential here.
mark On Wed, Jul 28, 2010 at 10:29 AM, Robbie Shepherd <[email protected]> wrote: > Simon, yes they are. > > Julio, I'm getting the same error on both os x and my server running Ubuntu > > Robbie > > On Wed, Jul 28, 2010 at 10:28 AM, Simon Russell <[email protected]> > wrote: >> >> Are they all coming from the same server? >> >> On Wed, Jul 28, 2010 at 10:25, Mark Wotton <[email protected]> wrote: >> > On three URLs? Surely not... >> > >> > mark >> > >> > On Wed, Jul 28, 2010 at 10:24 AM, Julio Cesar Ody <[email protected]> >> > wrote: >> >> I can't really tell from that stacktrace if the problem is this, but >> >> you may be hitting the maximum number of open files (and sockets). >> >> >> >> Are you on OSX? If so, you can expand that limit with sysctl. Try: >> >> >> >> sudo sysctl -w kern.maxfiles=36000 >> >> >> >> Sets to a bit over twice as much the default. Also, check >> >> kern.maxfilesperproc. And ulimit. >> >> >> >> >> >> >> >> >> >> On Wed, Jul 28, 2010 at 10:15 AM, Robbie Shepherd >> >> <[email protected]> wrote: >> >>> >> >>> Does anyone know if there's a maximum number of open connections >> >>> allowed when using hpricot/net::http? >> >>> >> >>> I have a bunch of xml feeds I need to read via hpricot and write to >> >>> disk. Whenever I hit it too hard/fast, or when I try to open several >> >>> files >> >>> in hpricot in a loop, my app dies. >> >>> >> >>> If I do something like... >> >>> >> >>> arr = [url1, url2, url2] >> >>> arr.each do|d| >> >>> doc = Hpricot(open(d)) >> >>> doc.search(blah...) >> >>> File.open("#{RAILS_ROOT}/xml/#{file_name}.xml", 'w') {|f| f.write >> >>> doc } >> >>> end >> >>> >> >>> I get: >> >>> >> >>> OpenURI::HTTPError: 500 Internal Server Error from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:277:in >> >>> `open_http' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:616:in >> >>> `buffer_open' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:164:in >> >>> `open_loop' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >> >>> `catch' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:162:in >> >>> `open_loop' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:132:in >> >>> `open_uri' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:518:in >> >>> `open' >> >>> from >> >>> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/open-uri.rb:30:in >> >>> `open' >> >>> from (irb):11 >> >>> from (irb):11:in `each' >> >>> from (irb):11 >> >>> >> >>> Am I missing something here, or is there a better way of doing this? >> >>> I'm on Ruby 1.8.7 and hpricot 0.8.2 >> >>> >> >>> Thanks >> >>> Robbie >> >>> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> >>> Groups "Ruby or Rails Oceania" group. >> >>> To post to this group, send email to [email protected]. >> >>> To unsubscribe from this group, send email to >> >>> [email protected]. >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> >> >> >> >> >> -- >> >> http://crazyhollywood.org >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Ruby or Rails Oceania" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/rails-oceania?hl=en. >> >> >> >> >> > >> > >> > >> > -- >> > A UNIX signature isn't a return address, it's the ASCII equivalent of a >> > black velvet clown painting. It's a rectangle of carets surrounding a >> > quote from a literary giant of weeniedom like Heinlein or Dr. Who. >> > -- Chris Maeda >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Ruby or Rails Oceania" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/rails-oceania?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby or Rails Oceania" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/rails-oceania?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > -- A UNIX signature isn't a return address, it's the ASCII equivalent of a black velvet clown painting. It's a rectangle of carets surrounding a quote from a literary giant of weeniedom like Heinlein or Dr. Who. -- Chris Maeda -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
