[
https://issues.apache.org/jira/browse/NUTCH-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943637#comment-14943637
]
Sebastian Nagel commented on NUTCH-2132:
----------------------------------------
No question, this is a significant improvement over FetchNodeDb (NUTCH-2011). A
few comments (without setting up a RabbitMQ consumer):
- (same as for FetchNodeDb) the default should be that nothing is done if no
publisher is configured. Even constructing a FetcherThreadEvent (it's a huge
object with a parsing fetcher) would mean a certain overhead
- since it's not trivial to configure the message queue, it's important to
catch improper or missing configurations early (in setConf()), and not when the
crawler is running:
{noformat}
fetch of ... failed with: java.lang.NullPointerException
at
org.apache.nutch.tools.RabbitMQPublisher.publish(RabbitMQPublisher.java:70)
at
org.apache.nutch.fetcher.FetcherThreadPublisher.publish(FetcherThreadPublisher.java:43)
at org.apache.nutch.fetcher.FetcherThread.run(FetcherThread.java:253)
{noformat}
- is it really necessary to send 2 events (start + end) per fetched URL?
Fetching is usually quite fast (< 1s). Right, it may be useful to track
documents which take long to fetch, but that's happening rarely and there are
timeouts to avoid that fetcher itself hangs forever.
- start and end event are on different loop levels (do-while): with redirects
and http.redirect.max > 0 you'll get unpaired events.
> Publisher/Subscriber model for Nutch to emit events
> ----------------------------------------------------
>
> Key: NUTCH-2132
> URL: https://issues.apache.org/jira/browse/NUTCH-2132
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, REST_api
> Reporter: Sujen Shah
> Labels: memex
> Fix For: 1.11
>
> Attachments: NUTCH-2132.patch
>
>
> It would be nice to have a Pub/Sub model in Nutch to emit certain events (ex-
> Fetcher events like fetch-start, fetch-end, a fetch report which may contain
> data like outlinks of the current fetched url, score, etc).
> A consumer of this functionality could use this data to generate real time
> visualization and generate statics of the crawl without having to wait for
> the fetch round to finish.
> The REST API could contain an endpoint which would respond with a url to
> which a client could subscribe to get the fetcher events.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
