Hi All,
I wanted to check in to see if anyone has found an answer for this
issue. I am injecting new URLs on a daily basis, and only need to
fetch/index those new one's, but obviously need to maintain a complete
webdb.
One thing I was thinking was to use a temporary webdb for the initial
injection, then updating (updatedb) my primary webdb after the fetch
or indexing.
# prepare dirs and inject urls
rm -rf $db/*
$nutch admin -local $db -create
$nutch inject -local $db -urlfile $urlFile
echo -e "\nGenerating next segment to fetch"
$nutch generate -local $db $segmentdir $fetchLimit
s=`ls -d $segmentdir/* | tail -1`
echo -e "\nFetching next segment"
$nutch fetch $s
echo -e "\nUpdating web database"
$nutch updatedb $dbmain $s
echo -e "\nAnalyzing links"
$nutch analyze $dbmain 5
OR after the segment is indexed -- as the above method wouldn't allow
a depth greather than 1?
# prepare dirs and inject urls
rm -rf $db/*
$nutch admin -local $db -create
$nutch inject -local $db -urlfile $urlFile
for i in `seq $depth`
do
echo -e "\nGenerating next segment to fetch"
$nutch generate -local $db $segmentdir $fetchLimit
s=`ls -d $segmentdir/* | tail -1`
echo -e "\nFetching next segment"
$nutch fetch $s
echo -e "\nUpdating web database"
$nutch updatedb $db $s
echo -e "\nAnalyzing links"
$nutch analyze $db 5
done
echo -e "\nFetch done"
echo "Indexing segments"
for s in `ls -1d $segmentdir/*`
do
$nutch index $s
done
echo -e "\nUpdating web database"
$nutch updatedb $dbmain $s
OR maybe I have no idea what I'm talking about : ) - I'm not a
developer, just trying to figure things out.
If anyone has experience with this and some advice I'm all ears. thanks!
Scott
On 11/10/05, Dean Elwood <[EMAIL PROTECTED]> wrote:
> Hi Lawrence,
>
> I'm stuck in the same position. I haven't yet examined the "merge" function,
> which might shed some light on it.
>
> Have you managed to discover anything so far?
>
> >>You can use the regular expression bases url filter. Than only urls that
> >>match the pattern will be added to a fetch list.<<
>
> Hi Stefan. Getting the new URL's to crawl is the easy part ;-)
>
> The trick, and the question, is how you add that to an existing database,
> and then re-index, without doing a full re-crawl?
>
> Thanks,
>
> Dean
>
> ----- Original Message -----
> From: "Lawrence Pitcher" <[EMAIL PROTECTED]>
> To: <[email protected]>
> Sent: Thursday, November 10, 2005 5:05 PM
> Subject: How to add only new urls to DB
>
>
> Hi,
>
> Thanks to all for the best search solution available.
>
> I have installed the software, indexed 15,000 websites and tested the search
> and it works great!
>
> If I want to add only two more websites, so I made a "newurls.txt" file,
> then injected it to WebDB "bin/nutch inject db/ -urlfile newurls.txt", then
> generated a new segment "bin/nutch generate db/ segments/", I then checked
> for the new sement name in the directory "/segments'
>
> Took that new segment name and placed it in the fetch command "bin/nutch
> fetch segments/20051110103316/"
>
> However it appears to re-fetch all 15,000 webpages along with the
> newurls.txt webpages.
>
> Can I not just index only the new and then Update the DB.
>
> Sorry for such a lame question but I have just started.
>
> Many thanks to all.
> Lawrence
>
>
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
