Hi user@,

I have been working on first-class Apache Nutch support in Atlas so that a
crawl can show up as catalog and lineage, without turning every URL into an
Atlas entity.

JIRA: https://issues.apache.org/jira/browse/ATLAS-5399
PR: https://github.com/apache/atlas/pull/748

The problem: Nutch crawls can be large. If Atlas stored one entity per URL,
a modest crawl would swamp the catalog. Operators still need to answer
basic questions: which crawl produced this index, which hosts were fetched,
which segment was generated, and how did indexing relate to the crawl?

What Atlas records: The new Nutch model keeps a bounded set of types:

   - nutch_crawl - the CrawlDb (crawl id + cluster)
   - nutch_seedlist - the inject seed directory
   - nutch_segment - each segment directory
   - nutch_domain - registrable domain (eTLD+1), e.g. apache.org
   - nutch_host - hostname, unique per Atlas cluster
   - nutch_index_process - one Atlas Process per Nutch IndexingJob
   - a generic DataSet - the search collection / index you wrote to


Fetch counts live on the crawl-to-host relationship (nutch_crawl_hosts),
not as a forest of page entities. There is no nutch_url type.

How metadata gets into Atlas: There are two writers on purpose:

   1. Atlas import bridge (the above ticket). hook-bin/import-nutch.sh
   reads CrawlDb (the primary Nutch database), optional HostDB, seeds, and
   segments. Hosts come from HostDB when you have run UpdateHostDb (a Nutch
   tool); otherwise they are rolled up from CrawlDb.
   2. Nutch Atlas IndexWriter (NUTCH-3210). After you index, Nutch emits
   one nutch_index_process whose inputs are the crawl/segment and whose output
   is the index DataSet. Lineage in the Atlas UI is then crawl + segment ->
   process -> index.


Both sides must use the same atlas.cluster.name and crawlId.

A short walkthrough

After a crawl (HostDB update recommended):

$ATLAS_HOME/hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> [-H
<hostdb>] [-s <seedDir>] [-g <segmentsDir>]

Then run your usual Nutch index job with indexer-atlas enabled. In the
Atlas UI you should see Nutch types in the entity tree and a lineage graph
on the indexing process.

If you crawl with Nutch and already run Atlas, review comments on the JIRA
or PR are welcome. The companion Nutch change is
https://issues.apache.org/jira/browse/NUTCH-3210
I'm polishing off that PR soon.

Thanks,
Lewis
http://people.apache.org/keys/committer/lewismc

Reply via email to