Hi dev@, I opened ATLAS-5399 to catalog Apache Nutch crawls in Atlas without creating one entity per URL.
JIRA: https://issues.apache.org/jira/browse/ATLAS-5399 PR: https://github.com/apache/atlas/pull/748 What this adds -------------- - Nutch types in addons/models/7000-Nutch (crawl, seedlist, segment, domain, host, nutch_index_process). Fetch metrics live on nutch_crawl_hosts. Hosts are global per cluster: {hostname}@ {clusterName}. - A batch import bridge (addons/nutch-bridge). Hosts come from Nutch HostDB when present, otherwise CrawlDb rollup. Seeds and segments are imported from the crawl directories. - Distro nutch-hook package and hook-bin/import-nutch.sh - Docs: docs/src/documents/Hook/HookNutch.md Indexing lineage is a split with Nutch. This PR defines nutch_index_process and the generic index DataSet shape; NUTCH-3210 (Nutch indexer-atlas) will emit one process per IndexingJob. The two sides share atlas.cluster.name and crawlId. Why this shape -------------- A large crawl would explode Atlas if every URL were an entity. Cataloging crawls, segments, domains, and hosts keeps the graph bounded and still gives lineage from crawl/segment to index process to search collection. How to try it ------------- Build Atlas, start the docker stack (postgres backend is enough), then: hook-bin/import-nutch.sh -c <crawlId> -d <crawldb> \ [-H <hostdb>] [-s <seedDir>] [-g <segmentsDir>] I ran this against a live docker Atlas with seed https://nutch.apache.org/ and confirmed crawl/seed/segment/domain/host entities plus lineage once the Nutch IndexWriter ran. Screenshot is on the PR. Reviews and comments on the JIRA or PR are welcome. A companion Nutch change is NUTCH-3210. I will submit that PR soon after some refinements. Thanks, lewismc http://people.apache.org/keys/committer/lewismc
