Hi user@, I have been working on first-class Apache Nutch support in Atlas so that a crawl can show up as catalog and lineage, without turning every URL into an Atlas entity.
JIRA: https://issues.apache.org/jira/browse/ATLAS-5399 PR: https://github.com/apache/atlas/pull/748 The problem: Nutch crawls can be large. If Atlas stored one entity per URL, a modest crawl would swamp the catalog. Operators still need to answer basic questions: which crawl produced this index, which hosts were fetched, which segment was generated, and how did indexing relate to the crawl? What Atlas records: The new Nutch model keeps a bounded set of types: - nutch_crawl - the CrawlDb (crawl id + cluster) - nutch_seedlist - the inject seed directory - nutch_segment - each segment directory - nutch_domain - registrable domain (eTLD+1), e.g. apache.org - nutch_host - hostname, unique per Atlas cluster - nutch_index_process - one Atlas Process per Nutch IndexingJob - a generic DataSet - the search collection / index you wrote to Fetch counts live on the crawl-to-host relationship (nutch_crawl_hosts), not as a forest of page entities. There is no nutch_url type. How metadata gets into Atlas: There are two writers on purpose: 1. Atlas import bridge (the above ticket). hook-bin/import-nutch.sh reads CrawlDb (the primary Nutch database), optional HostDB, seeds, and segments. Hosts come from HostDB when you have run UpdateHostDb (a Nutch tool); otherwise they are rolled up from CrawlDb. 2. Nutch Atlas IndexWriter (NUTCH-3210). After you index, Nutch emits one nutch_index_process whose inputs are the crawl/segment and whose output is the index DataSet. Lineage in the Atlas UI is then crawl + segment -> process -> index. Both sides must use the same atlas.cluster.name and crawlId. A short walkthrough After a crawl (HostDB update recommended): $ATLAS_HOME/hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> [-H <hostdb>] [-s <seedDir>] [-g <segmentsDir>] Then run your usual Nutch index job with indexer-atlas enabled. In the Atlas UI you should see Nutch types in the entity tree and a lineage graph on the indexing process. If you crawl with Nutch and already run Atlas, review comments on the JIRA or PR are welcome. The companion Nutch change is https://issues.apache.org/jira/browse/NUTCH-3210 I'm polishing off that PR soon. Thanks, Lewis http://people.apache.org/keys/committer/lewismc
