This isn't really a great match for map/reduce programming which is mostly what hadoop is intended to facilitate.
What you are looking for is something more along the lines of Amazon's simple queue service. That said, the hadoop distributed file service (HDFS) and the associated clone of bigtables (hbase) might be very interesting to you. HDFS comes with hadoop and hbase will soon come with hadoop. I haven't used it, but it appears to me that hbase isn't quite ready for prime time, but it is making fast strides. You should consider carefully whether Hadoop is right for you relative to your needs for reliability. It is new software and that always increases risk even if you have somebody like Doug Cutting leading the effort. On 11/18/07 6:49 AM, "Eugeny N Dzhurinsky" <[EMAIL PROTECTED]> wrote: > Hello there! > > We are looking forward to use hadoop for our distributed system, however > it's still unclear for us, is it possible to solve such tasks: > > 1) maintain fixed number of concurrent jobs executed at the same time > > 2) grant access for all jobs to the same static resource, which they should > share. This resource is mostly FIFO queue, which is filled by third-party > application, and could be filled by certain jobs if they met some conditions. > So the job can put data into this queue, which needs to be picked by this job > on text iteration or some another job > > 3) schedule new jobs when some of previously scheduled jobs completes, or > allows the jobs to check for new data in FIFO queue described above > > Thank you in advance!
