Couldn't look at your code yet, but: AFAIK Seda was developed for asynchronous I/O, which would mean a redesign of the central FetcherTask class. If every thread downloads 50 files at once, you only need a couple of them in parallel to saturate the network interfaces.
Clemens ----- Original Message ----- From: "David Worms" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: "Avalon users" <[EMAIL PROTECTED]> Sent: Monday, February 17, 2003 11:32 PM Subject: [LARM] using SEDA, pros and cons > > In order to move our project to the next level, it has been discussed > the use of an SEDA architecture. > > With the help of the Excalibur / Event library, I created a simplified > version of Larm which use SEDA as its backbone architecture. The goal > was not to make it pretty or even working, but to see how each > application components could feet in an event stage architecture. > > The great thing I notice is how flexible the application become. it is > extremely easy to map each stage with the others. One of the features > of LARM is the ability to have different sources( db, web, filesystem, > ... ), process them, and store them (lucene index, log, ...). This > seems easily achieveable with SEDA. Also clustering the LARM could be > done through a specific stage implementation. > > This was for the pros. However, it will be great to get some feedback > because I am really not sure on how to deal with SEDA. here is some > problems I am facing. > > - In order for the crawler to be efficient, I had to raise the number > of threads, but from what I read in the past, only one or two threads > should be used in a SEDA environment. > > - Also, it looks to consume a lot of memory, which could be due to the > number of messages put into the queue. > > Clemens, and others, please have a look of it, and give me some > feedback. > > http://67.116.155.180/~wdavidw/stage.zip > > David. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
