[ http://issues.apache.org/jira/browse/HADOOP-490?page=comments#action_12435209 ] Andrzej Bialecki commented on HADOOP-490: ------------------------------------------
(Oops, I thought JIRA would include the link in the comment). NUTCH-368 provides an implementation of a filesystem-based message queue system. This implementation is not Nutch specific, and can be easily moved to Hadoop if users find it useful. > Add ability to send "signals" to jobs and tasks > ----------------------------------------------- > > Key: HADOOP-490 > URL: http://issues.apache.org/jira/browse/HADOOP-490 > Project: Hadoop > Issue Type: New Feature > Components: mapred > Affects Versions: 0.6.0 > Reporter: Andrzej Bialecki > > In some cases it would be useful to be able to "signal" a job and its tasks > about some external condition, or to broadcast a specific message to all > tasks in a job. Currently we can only send a single pseudo-signal, that is > to kill a job. > Example 1: some jobs may be gracefully terminated even if they didn't > complete all their work, e.g. Fetcher in Nutch may be running for a very long > time if it blocks on relatively few sites left over from the fetchlist. In > such case it would be very useful to send it a message requesting that it > discards the rest of its input and gracefully completes its map tasks. > Example 2: available bandwidth for fetching may be different at different > times of day, e.g. daytime vs. nighttime, or total external link usage by > other applications. Fetcher jobs often run for several hours. It would be > good to be able to send a "signal" to the Fetcher to throttle or un-throttle > its bandwidth usage depending on external conditions. > Job implementations could react to these messages either by implementing a > method, or by registering a listener, whichever seems more natural. > I'm not quite sure how to go about implementing it, I guess this would have > to be a part of TaskUmbilicalProtocol but my knowledge here is a bit fuzzy > ... ;) Comments are welcome. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
