I agree. The patch has been applied to Nutch 1.2 and 2.0, so it is in the loop. However, my understanding is the timeout patch won't fix the compress tika issue. And I know it didn't for me.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of AJ Chen Sent: Wednesday, August 04, 2010 10:56 AM To: [email protected] Subject: Re: Nutch Parser: Tika hangs on corrupt zip files fix due soon I also found tika hangs up occasionally on other content types. Before tika becomes reliable (meaning: don't just hangup unexpectedly but quit when it can't handle a doc), I would like to see the parser timeout patch gets into new nutch release. I have applied the timeout mechanism to prevent from the headache. -aj On Wed, Aug 4, 2010 at 10:18 AM, brad <[email protected]> wrote: > > With Nutch 1.1 I have been experiencing consistent hangs on zip files. > I have had as many as 15 errant threads in one Nutch process. The > issue is caused by Tika (TIKA-401 > https://issues.apache.org/jira/browse/TIKA-401) > or > more accurately by a library Tika uses Commons Compress aka > commons-compress-1.0.jar (issue compress-87 > https://issues.apache.org/jira/browse/COMPRESS-87 ). The issue is > being fixed in Commons Compress 1.1. Commons Compress 1.1 has not > been released yet, but is expected to be released any day now. > > > To see if Commons Compress 1.1 fixes the problem, I downloaded the > current commons compress-1.1 and built it. > Then I renamed .../nutch/plugins/parse-tika/commons-compress-1.0.jar > to commons-compress-1.0.jar_orig Then I copied > commons-compress-1.1-SNAPSHOT.jar to > .../nutch/plugins/parse-tika/commons-compress-1.1.jar > > I have run 5 large fetch and parses and I have not had a single hang > of tika parser on zip files and CPU utilization has dropped in half... > > So it looks like updated Commons Compress 1.1 will solve problem. > > So hopefully this will make into the next Tika release which will make > it into Nutch 1.2. > > > > > -- AJ Chen, PhD Chair, Semantic Web SIG, sdforum.org http://web2express.org twitter @web2express Palo Alto, CA, USA
