On Monday, October 28, 2013, sebb wrote: > On 28 October 2013 01:26, sebb <seb...@gmail.com <javascript:;>> wrote: > > On 26 October 2013 20:23, Philippe Mouawad > > <philippe.moua...@gmail.com<javascript:;>> > wrote: > >> Hello, > >> These days Cloud based testing is becoming popular and having to > distribute > >> test data accross many servers through CSV can become painful if not > >> impossible. > >> > >> Even without Cloud, when using distributed testing you always have to > >> replicate your data on all servers, which is a painful manual step. > >> > >> Shouldn't we introduce a new DataSet more suitable for these use cases ? > >> > >> We could do it in many different ways: > >> - Integrate an automatic CSV replicator, this would remain simple and > would > >> not introduce any tier, but with cloud based it would not horizontally > >> scale easily > >
> > Not sure I follow the scaling argument. The file would only have to be > > copied once before the test proper starts. > > From then on, the data is accessed locally. > > The scale word was not good, In my thinking the issue is more about deploying/copying/splitting the file among nodes or cloud members. A centralized access makes it far easier. > > However, with a database, each node will need at least one connection > > to the database, and every time more data is needed there will be > > network traffic. > > Or the database has to be running on the server node. > > Agree, I was not saying anything different. But as I said this can be useful for middle or low volume > >> - Use a JDBC DataSet, but we would need to ensure it performs fine, and > >> jdbc protocol is not well suited for cloud based deployment (But it > could > >> also be an interesting feature for Continuous Integration) > >> - Use a NOSQL repository (Redis seems to me the best choice) , see this > >> high level summary which I find interesting > >> > http://www.journaldunet.com/developpeur/outils/comparatif-des-bases-nosql/comparatif-des-bases-nosql-tableau-de-synthese.shtml > >> > >> I have implemented a new Redis (based on Java library for Redis) DataSet > >> which I plan to commit if no objection. > >> > >> It will introduce 2 new dependencies with Apache License: > >> - Jedis (http://code.google.com/p/jedis/) > >> - commons-pool > > > > There is also a dependency on Redis, but I guess that would not be > bundled. > > no need to anything else than jedis + commons-pool > I've just noticed that Redis is provided as source which needs to be > built before use. > If it's difficult to copy CSV files to cloud nodes, it seems to me > it's going to be much harder to install Redis. > In which case additional network traffic will be needed to access the > database. > > Also there is no official Windows release. > > >> Thoughts ? > > > > Is MongoDB not suitable? > > We already include a jar for it. > > Mongodb is more document oriented and has another type of use cases. One interesting feature of redis is lists where you can pop a line it will not be available to next calls, it is suitable for tests that need uniqueness accross all nodes. > >> -- > >> Regards. > >> Philippe M. > >> @philmdot <https://twitter.com/philmdot> > -- Cordialement. Philippe Mouawad.
