On Philippe's request... I'm not at computer anymore, so I'll answer properly tomorrow...
BUT, it sounds odd that S3 should be slower than a "home made" layer across the Amazon network... what comes to mind, why isn't such layer already present transparently? I can see only three reasons; 1. Something in your conclusions are wrong, 2. Amazon developers ain't smart enough, 3. Amazon intentionally want it to be slow. Anything else you can think of? -- Cheers Niclas ---------- Forwarded message ---------- From: "philippe van dyck" <[email protected]> Date: 30 Nov 2009 20:27 Subject: Re: Code for S3 ES To: "Niclas Hedhman" <[email protected]> We dropped off the list, I don't mind going back on it (I let you post this back) Le 30 nov. 2009 à 12:27, Niclas Hedhman a écrit : > On Mon, Nov 30, 2009 at 4:54 PM, philippe van dyck <[email protected]> wrote: > >> Was my first s... Indeed, if you don't manage the sequence yourself and let the jclouds library and httpnio manage the threads you may loose the order needed to apply the new "layer" of updates to your entity store. Those operations, even a read operation, are ordered. That is the reason I use a queue. Accessing S3 is quite slow, and I tried previously to access it synchronously... and it was much too slow (I got timeouts in Wicket page display). So I definitely needed an asynchronous way to access S3, that is why the "cache" was added. > >> So at first, I though about using the underlying library but I also needed to implement a lo... Bad news, you simply cannot lock entries in S3 (no way to create a semaphore) ... you need a cluster-wide mechanism to do so (sounds like some NASA project). > >> I preferred to use one queue of updates / key. It is simple but not very efficient regarding ... Well, I had a stupid but efficient idea... what about synchronizing MapEntityStore.applyChanges ? Then I had a better one : final public BlockingQueue<BlockingQueue<T>> pipe As long as updates are sent *in the right order* asynchronously, I don't mind queueing the 'updates queues' in a pipe. Sound a bit overkill... but it works ! One more thing, when I empty a queue of updates (containing read operations with waiting threads), I do it in parallel, using a thread / entity reference. To be sure we both understand each other, I am talking about a pipe containing queues of updates to be applied one after the other, in parallel if the entity references are different. > >> Another added value of Infinispan is that it has it's own transaction manager (XA!)... I plan... Well, the XA stuff is managed cluster-wide by infinispan (don't ask, looks like a big locking mess - JGroups stuff) > >> Agreed, caching should be available as a generic mechanism elsewhere and used in the ES the '... We may need slowly but surely to stop talking about 'caching' ... since we actually use it to store indefinitely stuff 'à la' NoSQL ? WDYT ? > >> I am right now trying to understand how Infinispan manage failures .... and I don't like a sy... No real alternative on the market right now. Quite easy to understand once you played with it. (Cloud diving is a new sport) Be warned, this is a Ringgit eating machine ;-) Phil > > > Cheers > -- > Niclas Hedhman, Software Developer > http://www.qi4j.org- > > > New Energy for Jav...
_______________________________________________ qi4j-dev mailing list [email protected] http://lists.ops4j.org/mailman/listinfo/qi4j-dev
