I should add something. I realize that before OOM it says: Skipping http://www.mg.co.za/; different batch id (null)
and it hangs a liitle and throws error. (During crawl process it was trying to do something with that website and after a hang time it was skipping it but sometimes later it try to do something with that web site and delayed there and throw OOM) 2013/3/24 Furkan KAMACI <[email protected]> > I run that command: > > cat /proc/24277/status | grep Threads > > 24277 is my pid. And thread count is increasing, now it is 635. Is that > usual? > > > 2013/3/24 Furkan KAMACI <[email protected]> > >> I have: Hbase: 0.90.5 >> >> At Nutch: gora-hbase-0.2.1.jar >> >> When I use: ps auxh | wc -l I get ~ 290 threads running at >> crawling and indexing process. >> When I run ulimit -s I get: 10240 >> >> What do you suggest for debugging can you explain it more? Here says: >> http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html#wp1000929I >> think -Xss default value is 256 >> KB for me, I don't know it is enough or not. I run all things at my local >> computer, not on a cluster. >> >> >> >> 2013/3/24 lewis john mcgibbney [via Lucene] < >> [email protected]> >> >> On the thread you pointed to Sebastian provides some clues on how to >>> properly DEBUG the issue. >>> You can try to DEBUG the issue. By this I mean actually DEBUGGING it, >>> not >>> just setting logging to DEBUG and hoping for excellent results.. this >>> will >>> unfortunately not happen. >>> Can you please confirm your HBAse version and gora-* artifact >>> versioning? >>> Thanks >>> Lewis >>> >>> On Sat, Mar 23, 2013 at 6:31 PM, kamaci <[hidden >>> email]<http://user/SendEmail.jtp?type=node&node=4050815&i=0>> >>> wrote: >>> >>> > Here is a simiilar conversation: >>> > >>> > >>> http://lucene.472066.n3.nabble.com/Nutch-1-6-java-lang-OutOfMemoryError-unable-to-create-new-native-thread-td4044231.html >>> > >>> > By the way when I run depth 20 and topN 20 even with 1 thread I still >>> get >>> > same error sometimes. I've got that error I've posted with depth 20 >>> and top >>> > 20 and default options (10 threads) >>> > >>> > >>> > >>> > -- >>> > View this message in context: >>> > >>> http://lucene.472066.n3.nabble.com/waitForCompletion-Error-tp4050809p4050814.html >>> >>> > Sent from the Nutch - User mailing list archive at Nabble.com. >>> > >>> >>> >>> >>> -- >>> *Lewis* >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the >>> discussion below: >>> >>> http://lucene.472066.n3.nabble.com/waitForCompletion-Error-tp4050809p4050815.html >>> To unsubscribe from waitForCompletion Error, click >>> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4050809&code=ZnVya2Fua2FtYWNpQGdtYWlsLmNvbXw0MDUwODA5fDEyODM4MDc0Mg==> >>> . >>> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> > -- View this message in context: http://lucene.472066.n3.nabble.com/waitForCompletion-Error-tp4050809p4050865.html Sent from the Nutch - User mailing list archive at Nabble.com.
