So crawlId is equivalent to setting a different crawldb location, while batchId is the equivalent to the segments?
Basically, I need to be able to fetch and parse only the currently generated urls, not the entire database. I was doing that with a batch script previously by only fetching and parsing the newest segment. On Mon, Jul 30, 2012 at 11:46 AM, Ferdy Galema <[email protected]>wrote: > Hi, > > Just to avoid confusion: There are 2 concepts, namely batchId and crawlId. > The batchId is a subset within the same table. The table is determined by > crawlId. Not all stores adhere to crawlId, as this requires a specific > implementation. At least HBaseStore supports it. > > Not sure what the best way is to collect all crawlIds. You could always do > a listing of your database. I've made some improvments to NutchJob that > includes the crawlId within each jobname. That definitely helps when > managing multiple crawls. Will commit this soon. > > Ferdy. > > On Mon, Jul 30, 2012 at 5:28 PM, Lewis John Mcgibbney < > [email protected]> wrote: > > > Hi Bai, > > > > This is something which I've also wanted but haven't got around to > > sorting out. It's 'kinda' similar to the problem we addressed in > > NUTCH-1349 [0]. > > > > AFAIK the only solution I have is to hack a logging implementation > > which would provide the crawlId's in a more verbose manner, then you > > could pick them up from your log(s) output. Other than that the next > > step would be to introduce a dedicated tool for the job? > > > > Any other thoughts? > > > > Lewis > > > > [0] https://issues.apache.org/jira/browse/NUTCH-1349 > > > > On Mon, Jul 30, 2012 at 4:17 PM, Bai Shen <[email protected]> > wrote: > > > How do I check what crawlIds currently exist? Previously I could look > in > > > my segments directory to see what needed to be processed. > > > > > > Thanks. > > > > > > > > -- > > Lewis > > >
