Matt, How do you manage your tasks? Do you lauch them periodically or keep them somehow running and feed them data?
Vadim On Mon, Jun 23, 2008 at 21:54, Matt Kent <[EMAIL PROTECTED]> wrote: > We use Hadoop in a similar manner, to process batches of data in > real-time every few minutes. However, we do substantial amounts of > processing on that data, so we use Hadoop to distribute our computation. > Unless you have a significant amount of work to be done, I wouldn't > recommend using Hadoop because it's not worth the overhead of launching > the jobs and moving the data around. > > Matt > > On Tue, 2008-06-24 at 13:34 +1000, Ian Holsman (Lists) wrote: >> Interesting. >> we are planning on using hadoop to provide 'near' real time log >> analysis. we plan on having files close every 5 minutes (1 per log >> machine, so 80 files every 5 minutes) and then have a m/r to merge it >> into a single file that will get processed by other jobs later on. >> >> do you think this will namespace will explode? >> >> I wasn't thinking of clouddb.. it might be an interesting alternative >> once it is a bit more stable. >> >> regards >> Ian >> >> Stefan Groschupf wrote: >> > Hadoop might be the wrong technology for you. >> > Map Reduce is a batch processing mechanism. Also HDFS might be critical >> > since to access your data you need to close the file - means you might >> > have many small file, a situation where hdfs is not very strong >> > (namespace is hold in memory). >> > Hbase might be an interesting tool for you, also zookeeper if you want >> > to do something home grown... >> > >> > >> > >> > On Jun 23, 2008, at 11:31 PM, Vadim Zaliva wrote: >> > >> >> Hi! >> >> >> >> I am considering using Hadoop for (almost) realime data processing. I >> >> have data coming every second and I would like to use hadoop cluster >> >> to process >> >> it as fast as possible. I need to be able to maintain some guaranteed >> >> max. processing time, for example under 3 minutes. >> >> >> >> Does anybody have experience with using Hadoop in such manner? I will >> >> appreciate if you can share your experience or give me pointers >> >> to some articles or pages on the subject. >> >> >> >> Vadim >> >> >> > >> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> > 101tec Inc. >> > Menlo Park, California, USA >> > http://www.101tec.com >> > >> > >> > >
