Niclas, I finally took a look at the JTA support from infinispan, and it works great. So I got rid of all the queues of queues of pipes of stuff and ... reduced the code size to three files (+2 xml configs)!
Everything seems to work fine, but the performance is dismal (5 times slower than my previous version) when used outside of EC2. I launched four instances of my web app, and they all connected to my infinispan cluster (backed on S3) and displayed the same content on each page. After a lot of concurrent modifications on all the web app instances, I checked the results and I there was no problem (no concurrent modification, no error, nothing, amazing). It is slow, but working very well and ... scalable ! I plan to try this on EC2 later, in order to validate the usability, hoping that a local network will remove the current performance problem. phil Le 30 nov. 2009 à 13:41, Niclas Hedhman a écrit : > > On Philippe's request... > > I'm not at computer anymore, so I'll answer properly tomorrow... > > BUT, it sounds odd that S3 should be slower than a "home made" layer across > the Amazon network... what comes to mind, why isn't such layer already > present transparently? I can see only three reasons; > > 1. Something in your conclusions are wrong, > > 2. Amazon developers ain't smart enough, > > 3. Amazon intentionally want it to be slow. > > Anything else you can think of? > > -- Cheers > Niclas > >> ---------- Forwarded message ---------- >> From: "philippe van dyck" <[email protected]> >> Date: 30 Nov 2009 20:27 >> Subject: Re: Code for S3 ES >> To: "Niclas Hedhman" <[email protected]> >> >> We dropped off the list, I don't mind going back on it (I let you post this >> back) >> >> >> Le 30 nov. 2009 à 12:27, Niclas Hedhman a écrit : >> > On Mon, Nov 30, 2009 at 4:54 PM, philippe van dyck <[email protected]> >> > wrote: > >> Was my first s... >> >> Indeed, if you don't manage the sequence yourself and let the jclouds >> library and httpnio manage the threads you may loose the order needed to >> apply the new "layer" of updates to your entity store. >> Those operations, even a read operation, are ordered. That is the reason I >> use a queue. >> >> Accessing S3 is quite slow, and I tried previously to access it >> synchronously... and it was much too slow (I got timeouts in Wicket page >> display). >> >> So I definitely needed an asynchronous way to access S3, that is why the >> "cache" was added. >> > >> So at first, I though about using the underlying library but I also >> > >> needed to implement a lo... >> >> Bad news, you simply cannot lock entries in S3 (no way to create a >> semaphore) ... you need a cluster-wide mechanism to do so (sounds like some >> NASA project). >> > >> I preferred to use one queue of updates / key. It is simple but not >> > >> very efficient regarding ... >> >> Well, I had a stupid but efficient idea... what about synchronizing >> MapEntityStore.applyChanges ? >> >> Then I had a better one : final public BlockingQueue<BlockingQueue<T>> pipe >> >> As long as updates are sent *in the right order* asynchronously, I don't >> mind queueing the 'updates queues' in a pipe. >> >> Sound a bit overkill... but it works ! >> >> One more thing, when I empty a queue of updates (containing read operations >> with waiting threads), I do it in parallel, using a thread / entity >> reference. >> >> To be sure we both understand each other, I am talking about a pipe >> containing queues of updates to be applied one after the other, in parallel >> if the entity references are different. >> > >> Another added value of Infinispan is that it has it's own transaction >> > >> manager (XA!)... I plan... >> >> Well, the XA stuff is managed cluster-wide by infinispan (don't ask, looks >> like a big locking mess - JGroups stuff) >> > >> Agreed, caching should be available as a generic mechanism elsewhere >> > >> and used in the ES the '... >> >> We may need slowly but surely to stop talking about 'caching' ... since we >> actually use it to store indefinitely stuff 'à la' NoSQL ? WDYT ? >> > >> I am right now trying to understand how Infinispan manage failures .... >> > >> and I don't like a sy... >> >> No real alternative on the market right now. >> >> Quite easy to understand once you played with it. (Cloud diving is a new >> sport) >> >> Be warned, this is a Ringgit eating machine ;-) >> >> Phil >> > > > Cheers > -- > Niclas Hedhman, Software Developer > http://www.qi4j.org >> > > > - New Energy for Jav... >> >> > > _______________________________________________ > qi4j-dev mailing list > [email protected] > http://lists.ops4j.org/mailman/listinfo/qi4j-dev
_______________________________________________ qi4j-dev mailing list [email protected] http://lists.ops4j.org/mailman/listinfo/qi4j-dev
