[wtr-general] Re: after massive watir crawl..
Curious, I'm not sure if your issue was resolved, but I wanted to let you know that I switched to WWW::Mechanize for a simple crawler that was hitting around 10,000 pages. The script finished in under an hour, verifying everything I needed. So, depending on your test cases and requirements, this might be a viable solution for you. The WWW::Mech version of my script was just over 8 lines; pretty simple. Adam On Sep 8, 11:12 am, AR reed.a...@gmail.com wrote: I've had the same issue while doing a little sanity checking on some marketing websites that we control. Using IE.new_process helped a little, but Ruby would always die before the script completed. On Aug 30, 6:33 pm, Chris McMahon christopher.mcma...@gmail.com wrote: looks to me that something must be stuck somewhere, and I need to clear that away, but I don't know what I need to clear away.. I restarted the computer, but the problem does not go away. Could anyone tell me what I need to clear away in order to have smooth internet access?? I seem to recall that Ruby under some circumstances will hog memory until Bad Things Happen. If you're crawling all these pages and saving bits of information, be sure you're writing out to a file along the way and flushing the write buffer when you do it. -Chris --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
[wtr-general] Re: after massive watir crawl..
I've had the same issue while doing a little sanity checking on some marketing websites that we control. Using IE.new_process helped a little, but Ruby would always die before the script completed. On Aug 30, 6:33 pm, Chris McMahon christopher.mcma...@gmail.com wrote: looks to me that something must be stuck somewhere, and I need to clear that away, but I don't know what I need to clear away.. I restarted the computer, but the problem does not go away. Could anyone tell me what I need to clear away in order to have smooth internet access?? I seem to recall that Ruby under some circumstances will hog memory until Bad Things Happen. If you're crawling all these pages and saving bits of information, be sure you're writing out to a file along the way and flushing the write buffer when you do it. -Chris --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
[wtr-general] Re: after massive watir crawl..
Well, if you're crawling tens of thousands of pages you might have picked up a nasty bug somewhere along the way. I would run some virus scans. Also run some network diagnostics (try pinging some sites, etc...) -Dylan On Aug 29, 8:04 am, curious csamigr...@gmail.com wrote: I have massive WATIR based crawling going on.. I am crawling many tens of thousands of pages. Now after this massive crawl, my internet is getting stuck. Internet is not loading the pages anymore.. and it is just saying 'loading' message, and not proceeding to the next page. I have cleared away / deleted all 'history / temporary files / cookies' in Internet browser - tools / options, but the problem still persists. I have tried the same WATIR crawling in another computer with the same internet connection, and in this another computer, internet access is just fine. But in this computer with massive crawling, internet pages are not loading. not only the pages I am crawling, but also any internet pages are not loading. looks to me that something must be stuck somewhere, and I need to clear that away, but I don't know what I need to clear away.. I restarted the computer, but the problem does not go away. Could anyone tell me what I need to clear away in order to have smooth internet access?? thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
