Hi Mike, My understanding is, in hadoop job scheduling is done implicitly as you said it spread load as much as possible. However, I want to control task assignments to nodes. Let me put in a context of ad-hoc networking application scenario where a mobile devices broadcast *Hello* packets periodically to keep informing their presence to neighbours and then through some algorithm they select a cluster head. Then cluster head assigns different tasks to different nodes.
Take another scenario of a distributed database where every node holds part of it (distribution of data should be done explicitly by the user, not by the hadoop implicit scheduling) and then apply a query on these nodes independently. Finally performing a merging of the results retrieved from distributed node databases. Is it possible to perform such kind of tasks using hadoop? Thanks a lot, -- Ahmad On Thu, Dec 17, 2009 at 11:11 AM, Mike Kendall <[email protected]> wrote: > it sounds to me like you might want to split what you want to do up > into two separate jobs entirely... i don't quite understand your use > case since the point of hadoop is to spread your load as much (and > haphazardly!) as possible. > > -mike > > On Wed, Dec 16, 2009 at 3:43 PM, Ahmad Ali Iqbal > <[email protected]> wrote: > > Hi All, > > > > I am interested to know that can we use hadoop for applications where > they > > need more control over the data and it can specify which node will do > which > > part of the processing or the storage. For instance, suppose that I have > two > > data files (datasets, say 1 and 2) and setup a hadoop with two datanodes > (A > > and B) in a distributed cache, can I specify dataset 1 should load on > node A > > and dataset 2 should load on node B? Also I have two tasks (a and b), is > it > > possible to perform task a on node A and task b on node B? > > > > In other words, I want to supply a pattern of operation file (specifying > > store and operation tasks) to hadoop to perform, will it be possible? If > it > > is, I would appreciate a link discussing this or if a sample > > code/application doing this. > > > > > > Thanks a lot, > > > > -- > > Ahmad > > >
