[
https://issues.apache.org/jira/browse/NUTCH-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943757#comment-14943757
]
Julien Nioche commented on NUTCH-2132:
--------------------------------------
Looking at it from a slightly different angle, couldn't you use Logstash to
aggregate and push to ElasticSearch? Most events are already present in the log
files. You'd be able to query ES (or any other backend supported by Logstash or
similar) for instance with Kibana and graph it all with 0 modifications to the
code and 0 overhead.
> Publisher/Subscriber model for Nutch to emit events
> ----------------------------------------------------
>
> Key: NUTCH-2132
> URL: https://issues.apache.org/jira/browse/NUTCH-2132
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, REST_api
> Reporter: Sujen Shah
> Labels: memex
> Fix For: 1.11
>
> Attachments: NUTCH-2132.patch
>
>
> It would be nice to have a Pub/Sub model in Nutch to emit certain events (ex-
> Fetcher events like fetch-start, fetch-end, a fetch report which may contain
> data like outlinks of the current fetched url, score, etc).
> A consumer of this functionality could use this data to generate real time
> visualization and generate statics of the crawl without having to wait for
> the fetch round to finish.
> The REST API could contain an endpoint which would respond with a url to
> which a client could subscribe to get the fetcher events.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
