Hahahahahahahaha -----Original message----- From: Tejas Patil<[email protected]> Sent: Friday 24th January 2014 16:20 To: [email protected] Subject: Fwd: [jira] [Commented] (NUTCH-356) Plugin repository cache can lead to memory leak
Hi Lewis, I wont be surprised if any user out there gets a cake for this jira ;) .. just like someone did over a MySQL bug (http://www.youtube.com/watch?v=oAiVsbXVP6k <http://www.youtube.com/watch?v=oAiVsbXVP6k>) Cheers !! ---------- Forwarded message ---------- From: Lewis John McGibbney (JIRA) <[email protected] <mailto:[email protected]>> Date: Fri, Jan 24, 2014 at 7:01 PM Subject: [jira] [Commented] (NUTCH-356) Plugin repository cache can lead to memory leak To: [email protected] <mailto:[email protected]> [ https://issues.apache.org/jira/browse/NUTCH-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880956#comment-13880956 <https://issues.apache.org/jira/browse/NUTCH-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880956#comment-13880956> ] Lewis John McGibbney commented on NUTCH-356: -------------------------------------------- Indeed Markus. This one is blast from the past for sure. I dont know if I was born when Nutch 0.8 was trunk ;) > Plugin repository cache can lead to memory leak > ----------------------------------------------- > > Key: NUTCH-356 > URL: https://issues.apache.org/jira/browse/NUTCH-356 > <https://issues.apache.org/jira/browse/NUTCH-356> > Project: Nutch > Issue Type: Bug > Affects Versions: 0.8 > Reporter: Enrico Triolo > Assignee: Markus Jelsma > Fix For: 2.3, 1.8 > > Attachments: ASF.LICENSE.NOT.GRANTED--NutchTest.java, > ASF.LICENSE.NOT.GRANTED--patch.txt, NUTCH-356-trunk.patch, cache_classes.patch > > > While I was trying to solve a problem I reported a while ago (see Nutch-314), > I found out that actually the problem was related to the plugin cache used in > class PluginRepository.java. > As I said in Nutch-314, I think I somehow force the way nutch is meant to > work, since I need to frequently submit new urls and append their contents to > the index; I dont (and I cant) have an urls.txt file with all urls Im going > to fetch, but I recreate it each time a new url is submitted. > Thus, I think in the majority of times you wont have problems using nutch > as-is, since the problem I found occours only if nutch is used in a way > similar to the one I use. > To simplify your test Im attaching a class that performs something similar to > what I need. It fetches and index some sample urls; to avoid webmasters > complaints I left the sample urls list empty, so you should modify the source > code and add some urls. > Then you only have to run it and watch your memory consumption with top. In > my experience I get an OutOfMemoryException after a couple of minutes, but it > clearly depends on your heap settings and on the plugins you are using (Im > using > protocol-file|protocol-http|parse-(rss|html|msword|pdf|text)|language-identifier|index-(basic|more)|query-(basic|more|site|url)|urlfilter-regex|summary-basic|scoring-opic). > The problem is bound to the PluginRepository singleton instance, since it > never get released. It seems that some class maintains a reference to it and > this class is never released since it is cached somewhere in the > configuration. > So I modified the PluginRepositorys get method so that it never uses the > cache and always returns a new instance (you can find the patch in > attachment). This way the memory consumption is always stable and I get no > OOM anymore. > Clearly this is not the solution, since I guess there are many performance > issues involved, but for the moment it works. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
