Yupe, thanks! --- wassalam, [bayu]
/sent from Android phone/ On Feb 2, 2014 10:51 PM, "Tejas Patil" <[email protected]> wrote: > On Sun, Feb 2, 2014 at 5:54 PM, Bayu Widyasanyata > <[email protected]>wrote: > > > Hi Tejas, > > > > It's works and great! :) > > After reconfigured and many times of generate, fetch, parse & update, the > > pages on 2nd level is being crawled. > > > > 1 question, Is it fine and correct if I modified my current > > crawler+indexing script into this pseudo (skeleton): > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > # example number of levels / depth (loop) > > LOOP=4 > > > > nutch->inject() > > > > loop[ =< $LOOP] > > { > > nutch->generate() > > nutch->fetch(a_segment) > > nutch->parse(a_segment) > > nutch->updatedb(a_segment) > > } > > > > nutch->solrindex() > > > > I don't think that this should be a problem. Remember to pass all the > segments generated in the crawl loop to the solrindex job using "-dir" > option. > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > Thank you! > > > > > > On Mon, Jan 27, 2014 at 3:46 AM, Bayu Widyasanyata > > <[email protected]>wrote: > > > > > OK I will apply it first and update the result. > > > > > > Thanks.- > > > > > > > > > On Sun, Jan 26, 2014 at 11:01 PM, Tejas Patil < > [email protected] > > >wrote: > > > > > >> Please copy this at the end (but above the end tag '</configuration>') > > in > > >> your $NUTCH/conf/nutch-site.xml: > > >> > > >> <property> > > >> <name>http.content.limit</name> > > >> <value>999999999</value> > > >> </property> > > >> > > >> <property> > > >> <name>http.timeout</name> > > >> <value>2147483640</value> > > >> </property> > > >> > > >> <property> > > >> <name>db.max.outlinks.per.page</name> > > >> <value>999999999</value> > > >> </property> > > >> > > >> Please check if the url got fetched correctly after every round: > > >> For the first round with seed as http://bappenas.go.id, after > > "updatedb" > > >> job, run these to check if they are into the crawldb. The first url > must > > >> be > > >> db_fetched while the second one must be db_unfetched: > > >> > > >> bin/nutch readdb <YOUR_CRAWLDB> -url http://bappenas.go.id/ > > >> bin/nutch readdb <YOUR_CRAWLDB> -url > > >> > > >> > > > http://bappenas.go.id/berita-dan-siaran-pers/kerjasama-pembangunan-indonesia-australia-setelah-pm-tony-abbot/ > > >> > > >> Now crawl for the next depth. After "updatedb"job, check if the second > > url > > >> got fetched using the same command again. ie. > > >> bin/nutch readdb <YOUR_CRAWLDB> -url > > >> > > >> > > > http://bappenas.go.id/berita-dan-siaran-pers/kerjasama-pembangunan-indonesia-australia-setelah-pm-tony-abbot/ > > >> > > >> Note that if there was any redirection, you need to look out the > target > > >> url > > >> in the redirection chain and use that url ahead for debugging. Verify > if > > >> the content you got for that url had text "Liberal Party" in the > parsed > > >> output using this command: > > >> > > >> bin/nutch readseg -get <LATEST_SEGMENT> > > >> > > >> > > > http://bappenas.go.id/berita-dan-siaran-pers/kerjasama-pembangunan-indonesia-australia-setelah-pm-tony-abbot/ > > >> > > >> For larger segments, you might get a OOM error. So in that case, take > > the > > >> entire segment dump using: > > >> bin/nutch readseg -dump <LATEST_SEGMENT> <OUTPUT> > > >> > > >> After all this is verified and everything looks good from the crawling > > >> side, run solrindex and check if you get the query results. If not, > then > > >> there was a problem while indexing the stuff. > > >> > > >> Thanks, > > >> Tejas > > >> > > >> > > >> On Sun, Jan 26, 2014 at 9:09 AM, Bayu Widyasanyata > > >> <[email protected]>wrote: > > >> > > >> > Hi, > > >> > > > >> > I just realized that my nutch didn't crawl the articles/pages (depth > > 2) > > >> > which shown on frontpage. > > >> > My target URL is: http://bappenas.go.id > > >> > > > >> > As shown on that frontpage (top right below the slider banners) > three > > >> is a > > >> > text link: > > >> > > > >> > "Kerjasama Pembangunan Indonesia-Australia Setelah PM Tony Abbot" > > >> > and its URL: > > >> > > > >> > > > >> > > > http://bappenas.go.id/berita-dan-siaran-pers/kerjasama-pembangunan-indonesia-australia-setelah-pm-tony-abbot/?&kid=1390691937 > > >> > > > >> > I tried to search with keyword "Liberal Party" (with quotes) which > > >> appear > > >> > on link (page) above but has no result :( > > >> > > > >> > Following is the search link queried: > > >> > > > >> > > > >> > > > http://bappenas.go.id/index.php/bappenas_search/result?q=%22Liberal+Party%22 > > >> > > > >> > I use individual script to crawl below: > > >> > > > >> > === > > >> > # Defines env variables > > >> > export JAVA_HOME="/opt/searchengine/jdk1.7.0_45" > > >> > export PATH="$JAVA_HOME/bin:$PATH" > > >> > NUTCH="/opt/searchengine/nutch" > > >> > > > >> > # Start by injecting the seed url(s) to the nutch crawldb: > > >> > $NUTCH/bin/nutch inject $NUTCH/BappenasCrawl/crawldb > > >> $NUTCH/urls/seed.txt > > >> > > > >> > # Generate fetch list > > >> > $NUTCH/bin/nutch generate $NUTCH/BappenasCrawl/crawldb > > >> > $NUTCH/BappenasCrawl/segments > > >> > > > >> > # last segment > > >> > export SEGMENT=$NUTCH/BappenasCrawl/segments/`ls -tr > > >> > $NUTCH/BappenasCrawl/segments|tail -1` > > >> > > > >> > # Launch the crawler! > > >> > $NUTCH/bin/nutch fetch $SEGMENT -noParsing > > >> > > > >> > # Parse the fetched content: > > >> > $NUTCH/bin/nutch parse $SEGMENT > > >> > > > >> > # We need to update the crawl database to ensure that for all future > > >> > crawls, Nutch only checks the already crawled pages, and only > fetches > > >> new > > >> > and changed pages. > > >> > $NUTCH/bin/nutch updatedb $NUTCH/BappenasCrawl/crawldb $SEGMENT > > -filter > > >> > -normalize > > >> > > > >> > # Indexing our crawl DB with solr > > >> > $NUTCH/bin/nutch solrindex > > >> > > http://localhost:8080/solr/bappenasgoid/$NUTCH/BappenasCrawl/crawldb > > >> > -dir< > > >> > > http://localhost:8080/solr/bappenasgoid/$NUTCH/BappenasCrawl/crawldb-dir > > >> >$NUTCH/BappenasCrawl/segments > > >> > === > > >> > > > >> > I run this script daily but it looks it never reach the single > article > > >> > pages which shown on the frontpage. > > >> > > > >> > If I read Tejas explained on another thread (shown below), should I > > two > > >> or > > >> > three times loops (generate -> fetch -> parse -> update) to produce > 2 > > >> or 3 > > >> > depth levels? > > >> > > > >> > QUOTES from Tejas' e-mail (subject: Questions/issues with nutch): > > >> > ***************** > > >> > On Sat, Jun 29, 2013 at 2:49 PM, Tejas Patil < > > [email protected] > > >> > >wrote: > > >> > Yes. Nutch would parse the HTML and extract the content out of it. > > >> Tweaking > > >> > around the code surrounding the parser would have made that happen. > If > > >> you > > >> > did something else, would you mind sharing it ? > > >> > > > >> > The "depth" is used by the Crawl class in 1.x which is deprecated in > > >> 2.x. > > >> > Use bin/crawl instead. > > >> > While running the "bin/crawl" script, the "<numberOfRounds>" option > is > > >> > nothing but the depth till which you want the crawling to be > > performed. > > >> > > > >> > If you want to use the individual commands instead, run generate -> > > >> fetch > > >> > -> parse -> update multiple times. The crawl script internally does > > the > > >> > same thing. > > >> > eg. If you want to fetch till depth 3, this is how you could do: > > >> > inject -> (generate -> fetch -> parse -> update) > > >> > -> (generate -> fetch -> parse -> update) > > >> > -> (generate -> fetch -> parse -> update) > > >> > -> solrindex > > >> > ***************** > > >> > > > >> > I also has commented line below on regex-urlfilter.txt file: > > >> > # skip URLs containing certain characters as probable queries, etc. > > >> > #-[?*!@=] > > >> > > > >> > Apps: nutch 1.7 and Solr 4.5.1 > > >> > > > >> > Thank you so much! > > >> > > > >> > -- > > >> > wassalam, > > >> > [bayu] > > >> > > > >> > > > > > > > > > > > > -- > > > wassalam, > > > [bayu] > > > > > > > > > > > -- > > wassalam, > > [bayu] > > >
