[ http://issues.apache.org/jira/browse/HADOOP-490?page=all ]
eric baldeschwieler updated HADOOP-490: --------------------------------------- -1 Unless I'm reading this wrong, a file per message would kill the name node at any scale. Also, in a large task, the cost of having every mapper/task scan all the messages could be fairly prohibitive. I'd suggest making it available in contrib or some other mechanism until we see how much uptake it gets. This would leave specific applications free to use it. Perhaps if this gains wide acceptance we could explore moving the concepts into core, but we would need to address the scaling issues to make a general facility. A very interesting set of ideas here, but very complicated if you want to make it work in large general cases. > Add ability to send "signals" to jobs and tasks > ----------------------------------------------- > > Key: HADOOP-490 > URL: http://issues.apache.org/jira/browse/HADOOP-490 > Project: Hadoop > Issue Type: New Feature > Components: mapred > Affects Versions: 0.6.0 > Reporter: Andrzej Bialecki > > In some cases it would be useful to be able to "signal" a job and its tasks > about some external condition, or to broadcast a specific message to all > tasks in a job. Currently we can only send a single pseudo-signal, that is > to kill a job. > Example 1: some jobs may be gracefully terminated even if they didn't > complete all their work, e.g. Fetcher in Nutch may be running for a very long > time if it blocks on relatively few sites left over from the fetchlist. In > such case it would be very useful to send it a message requesting that it > discards the rest of its input and gracefully completes its map tasks. > Example 2: available bandwidth for fetching may be different at different > times of day, e.g. daytime vs. nighttime, or total external link usage by > other applications. Fetcher jobs often run for several hours. It would be > good to be able to send a "signal" to the Fetcher to throttle or un-throttle > its bandwidth usage depending on external conditions. > Job implementations could react to these messages either by implementing a > method, or by registering a listener, whichever seems more natural. > I'm not quite sure how to go about implementing it, I guess this would have > to be a part of TaskUmbilicalProtocol but my knowledge here is a bit fuzzy > ... ;) Comments are welcome. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
