Thanks Chris! Appreciate your taking time to explain in detail. I think would like to first explore Terracota.
thanks ashish On Fri, Aug 28, 2009 at 10:20 PM, Christopher Popp<[email protected]> wrote: >> Couple of queries, >> 1. Each client getting its queue will be costly from >> management/implementation perspective. Like 120K queues maintained and >> each one taking care of its registered listener(s) >> 2. Was there any noticeable delay in processing, with this architecture >> 3. Will this work for protocols like SMPP, which sends a message and >> expects a response with an ID. (may be it could work, its just that >> the client needs to be a bit smarter) >> >> thanks >> ashish > > > 1.) The way Terracotta handles clustering of JVMs is useful to mitigate the > cost of this. The Terracotta server manages the objects that have been > "shared" by various JVMs in the cluster. Once an object is shared, it can > actually be flushed out of the JVMs and only stored on the Terracotta server. > The Terracotta server itself is capable of paging these objects out to disk. > So, even if a particular MINA system has 20,000 client connections with a > queue for each, a lot of them can actually be paged out of memory assuming > they are inactive. The ones that are needed are paged into the JVM as > necessary. (Terracotta does various operations to minimize traffic such as > sending diffs of objects, and uses caching strategies to attempt to keep > frequently hit objects in the JVM). So, this keeps memory usage due to the > queues to a minimum. > > Second...Terracotta works by the standard synchronization primitives you're > used to. To minimize the work that one of the MINA servers has to do to > listen to the queues, I tried to take advantage of this. I wrapped each > queue with a sort of observer/observable pattern. In front of this, I added > a sort of "select" like interface in the MINA servers. So...when the MINA > server is inactive, it is sitting there in the typical multithreaded manner > waiting for work. When one of the front end servers pushes a job onto a > queue, the MINA server is notified, which ends up pushing that queue on a > list of ready queues for that MINA server, and then triggers a notify. > > So...sure, it is somewhat costly to maintain queues and such, and you could > do it in a more efficient way perhaps depending on your exact requirements. > For our case, we wanted to avoid the possibility of Out of Memory exceptions > on our various cluster members, as well as provide for the option of queued > jobs which will be processed as possible by the MINA servers. With some of > the care we took, it doesn't really seem to put too much load on any of the > particular systems...and hopefully in cases where the systems are overloaded > by a burst of activity, they will be capable of working through it, just at a > reduced response time. > > 2.) With a ton of clients connected, sending a message destined for a single > client through the front end server (so, front end server --> terracotta > server --> MINA server, and back through), did not add much in the way of > delay. Message throughput on the tens or hundreds of milliseconds level per > message is not exactly critical to our application, so we're not exactly at > that level however. We have also done some testing where 10,000 to 20,000 > clients will be connected, and we send broadcast operations that are intended > to go to every client, and then mix in some targeted operations as well. > This can be kind of slow. Terracotta distributes lock ownership and > acquisition, and we have some locks that become highly contested when the > system gets busy with messages passing between multiple front end servers and > MINA backend servers. This is something we see as a great opportunity for > performance enhancement when it comes to it. Terracotta itself > offers pretty high levels of throughput. Their open source offering doesn't > support it, but they have a paid-for enterprise version that supports > striping of servers which brings the throughput much higher. > > 3.) Our system is setup so that operations come full circle and the response > ends up back at the front end server and returned to the caller. In short, > the process is kind of like this...FRONT END SERVER creates a JOB, the JOB is > then dispersed to queues for any clients it is destined for. (1 for a single > client operation, all queues for a broadcast, or anything in between). The > FRONT END SERVER does a wait() on the JOB. Meanwhile, BACKEND SERVERS have > been notified of JOBS put on the queues. They communicate however they need > to with their client, and then they put the response from the client back > into a mapping of client-->result inside the JOB. Once the number of results > matches the number of clients, a notify() occurs, and the FRONT END SERVER > now proceeds, pulling the results for the clients out of the JOB, and passing > the result back to whomever made the call to the FRONT END SERVER. It is > very much like a typical multithreaded > application, since the Terracotta JVM clustering shares object graphs and > synchronization primitives across the cluster. > > As an alternative to the synchronous approach above, we also allow for a JOB > to be submitted and a job ID immediately returned. This asynchronous > approach allows a client to potentially come back in and query the status of > a job in the system. This might be used when someone wants to submit an > operation that will be broadcasted to all clients, or some large number that > will take a while to process. > > I'm not sure how well it would work to try and create a generic approach that > would fit the needs of different protocols and such on top of MINA. Most of > the clustering happened a level above MINA, more in our application logic. > It worked well for us though since we have a custom protocol to implement > anyway, and weren't just picking up something like FTP and hoping for it to > automatically scale and be highly available with MINA. > > Chris Popp > > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
