Hi Chris https://issues.apache.org/jira/browse/NUTCH-1087
Use the org.apache.nutch.net.URLFilterChecker to test. cheers > Hey Markus, > > On Nov 24, 2011, at 9:50 AM, Markus Jelsma wrote: > > > Hey Markus, > > > > > > On Nov 24, 2011, at 9:16 AM, Markus Jelsma wrote: > > > > I think Marek is right, the crawl-filter _is_ used in the crawl > > > > command. I don't know what happens if it isnt there. > > > > > > Interesting. Where is the crawl-urlfilter.txt? It's not in my built > > > conf directory in runtime/local/conf from 1.4? > > > > Its gone! I checked and last saw it in 1.2. Strange > > > > > > Good reasons to get rid of the crawl command and stuff in 1.5 if you > > > > ask me. > > > > > > I'd be in favor of replacing the current Crawl command with a simple > > > Java driver that just calls the underlying Inject, Generate, and Fetch > > > tools. Would that work? > > > > There's an open issue to replace it with a basic crawl shell script. It's > > easier to understand and uses the same commands. Non-Java users should > > be able to deal with it better, and provide us with better problem > > descriptions. > > +1, that would be cool indeed. Do you know what issue it is? > > BTW, I'm currently instrument urlfilter-regex to see if I can figure out > if it's dropping the at_download URLs for whatever reason. Sigh. > > Cheers, > Chris > > > > Cheers, > > > Chris > > > > > > > On Thursday 24 November 2011 16:59:42 Mattmann, Chris A (388J) wrote: > > > >> Hi Marek, > > > >> > > > >> On Nov 24, 2011, at 7:46 AM, Marek Bachmann wrote: > > > >>> I think when you use the crawl command instead of the single > > > >>> commands, you have to specify the regEx rules in the > > > >>> crawl-urlfilter.txt file. But I don't know if it is still the case > > > >>> in 1.4 > > > >>> > > > >>> Could that be the problem? > > > >> > > > >> Doesn't look like there's a crawl-urlfilter.txt in the conf dir. > > > >> Also it looks like urlfilter-regex is the one that's enabled by > > > >> default and shipped with the basic config. > > > >> > > > >> Thanks for trying to help though. I'm going to figure this out! Or, > > > >> someone is going to probably tell me what I'm doing wrong. > > > >> We'll see what happens first :-) > > > >> > > > >> Cheers, > > > >> Chris > > > >> > > > >>> On 24.11.2011 16:20, Mattmann, Chris A (388J) wrote: > > > >>>> On Nov 24, 2011, at 3:21 AM, Julien Nioche wrote: > > > >>>>>> OK, nm. This *is* different behavior from 1.3 apparently, but I > > > >>>>>> figured out how to make it work in 1.4 (instead of editing the > > > >>>>>> global, top-level conf/nutch-default.xml, > > > >>>>>> I needed to edit runtime/local/conf/nutch-default.xml). Crawling > > > >>>>>> is forging ahead. > > > >>>>> > > > >>>>> yep, I think this is documented on the Wiki. It is partially why > > > >>>>> I suggested that we deliver the content of runtime/local as our > > > >>>>> binary release for next time. Most people use Nutch in local > > > >>>>> mode so this would make their lives easier, as for the advanced > > > >>>>> users (read pseudo or real distributed) they need to recompile > > > >>>>> the job file anyway and I'd expect them to use the src release > > > >>>> > > > >>>> +1, I'll be happy to edit build.xml and make that happen for 1.5. > > > >>>> > > > >>>> In the meanwhile, time to figure out why I still can't get it to > > > >>>> crawl the PDFs... :( > > > >>>> > > > >>>> Cheers, > > > >>>> Chris > > > >>>> > > > >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >>>> Chris Mattmann, Ph.D. > > > >>>> Senior Computer Scientist > > > >>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > > >>>> Office: 171-266B, Mailstop: 171-246 > > > >>>> Email: [email protected] > > > >>>> WWW: http://sunset.usc.edu/~mattmann/ > > > >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >>>> Adjunct Assistant Professor, Computer Science Department > > > >>>> University of Southern California, Los Angeles, CA 90089 USA > > > >>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >> > > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >> Chris Mattmann, Ph.D. > > > >> Senior Computer Scientist > > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > > >> Office: 171-266B, Mailstop: 171-246 > > > >> Email: [email protected] > > > >> WWW: http://sunset.usc.edu/~mattmann/ > > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > >> Adjunct Assistant Professor, Computer Science Department > > > >> University of Southern California, Los Angeles, CA 90089 USA > > > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > Chris Mattmann, Ph.D. > > > Senior Computer Scientist > > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > > Office: 171-266B, Mailstop: 171-246 > > > Email: [email protected] > > > WWW: http://sunset.usc.edu/~mattmann/ > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > Adjunct Assistant Professor, Computer Science Department > > > University of Southern California, Los Angeles, CA 90089 USA > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Senior Computer Scientist > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 171-266B, Mailstop: 171-246 > Email: [email protected] > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Assistant Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
