Nm, I figured it out. had to run the webapp command. There should be a wiki to document this. I could volunteer to write one, if nobody else is going to do this?
On Thu, Sep 25, 2014 at 10:23 PM, Nima Falaki <[email protected]> wrote: > Thanks Talat, just wondering is there a set of instructions that I can use > to get the nutch web admin tool up and running? > > Nima > > On Thu, Sep 25, 2014 at 9:37 PM, Talat Uyarer <[email protected]> wrote: > >> Hi Nima, >> >> I never used nutch web admin. Web admin that you used, is very old. Maybe >> you can use our brand new web admin development ( >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/NUTCH-841). >> Now >> it is just committed trunk and 2.x branches. >> >> For your question IMHO start URLs means your seedlist. For nutch accept as >> a folder or text file. >> >> Limit URLs mean when crawler start using seed list which new URL will be >> accepted for next steps. Actually you can right regex rules in nutch. For >> example you crawl a home page of news webpage but you want to only get >> sports urls. You can write a regex rule for accepting sports URLs. >> >> For reverse situations you can use Exclude URLs. >> >> Talat. >> Hello Everyone: >> >> I am following the directions exactly word for word in this tutorial >> >> https://github.com/101tec/nutch/wiki/admin-url-upload >> >> My question is what is the difference between the start and limit urls. >> From the wiki I saw that the limit url seems to be a flat list of urls we >> want to fetch, but then why have a start url to become with? >> >> Also I noticed that when you do not have a limit url, you get the >> following >> exception (Note: I am using nutch-gui-0.5-dev), >> >> When you start a crawl shouildnt there be some sort of pop up box that >> occurs pops up saying that you need a limit url? So you dont get this >> exception? I can help work on this. >> >> >> 14/09/25 20:34:37 INFO [Thread-3071] (Fetcher.java:970) - Fetcher: done >> >> 14/09/25 20:34:37 INFO [Thread-3071] (BWUpdateDb.java:217) - bw update: >> starting >> >> 14/09/25 20:34:37 INFO [Thread-3071] (BWUpdateDb.java:218) - bw update: >> db: /private/tmp/nutch/Nima/crawls/Crawl-2014.09.25_20.34.20/crawldb >> >> 14/09/25 20:34:37 INFO [Thread-3071] (BWUpdateDb.java:219) - bw update: >> bwdb: /private/tmp/nutch/Nima/crawls/Crawl-2014.09.25_20.34.20/bwdb >> >> 14/09/25 20:34:37 INFO [Thread-3071] (BWUpdateDb.java:220) - bw update: >> segments: >> >> [/private/tmp/nutch/Nima/crawls/Crawl-2014.09.25_20.34.20/segments/20140925203433] >> >> 14/09/25 20:34:37 INFO [Thread-3071] (BWUpdateDb.java:223) - bw update: >> wrapping started. >> >> 14/09/25 20:34:37 WARN [Thread-3071] (JobClient.java:547) - Use >> GenericOptionsParser for parsing the arguments. Applications should >> implement Tool for the same. >> >> 14/09/25 20:34:38 INFO [Thread-3071] (BWUpdateDb.java:248) - bw update: >> filtering started. >> >> 14/09/25 20:34:38 WARN [Thread-3071] (JobClient.java:547) - Use >> GenericOptionsParser for parsing the arguments. Applications should >> implement Tool for the same. >> >> 14/09/25 20:34:38 WARN [Thread-3071] (StartCrawlRunnable.java:57) - can >> not start crawl.14/09/25 20:34:38 WARN [Thread-3071] >> (StartCrawlRunnable.java:57) - can not start crawl. >> >> org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: >> file:/private/tmp/nutch/Nima/crawls/Crawl-2014.09.25_20.34.20/bwdb/current >> >> at >> >> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179) >> >> at >> >> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:39) >> >> at >> >> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190) >> >> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797) >> >> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142) >> >> at org.apache.nutch.crawl.bw.BWUpdateDb.update(BWUpdateDb.java:263) >> >> at org.apache.nutch.crawl.CrawlTool.crawl(CrawlTool.java:135) >> >> at >> >> org.apache.nutch.admin.crawl.StartCrawlRunnable.run(StartCrawlRunnable.java:52) >> >> at java.lang.Thread.run(Thread.java:744) >> >> >> -- >> >> >> >> Nima Falaki >> Software Engineer >> [email protected] >> > > > > -- > > > > Nima Falaki > Software Engineer > [email protected] > > -- Nima Falaki Software Engineer [email protected]
