Hi All, Are we use, Redis distributed setup? In a MB containerized Cluster (DOCKER), how can we ensure, the data remain when containers kill and start? MB is supporting kubernates cluster?
Thanks On Sun, Jul 16, 2017 at 9:41 AM, Wishmitha Mendis <[email protected]> wrote: > Hi Kevin, > > I just tried jedis for the initial implementation, no actual reason reason > to be honest. I will try other java-clients you mentioned, and figure out > what will fit the best for the project scenario. > > Best Regards, > > On Fri, Jul 14, 2017 at 11:17 PM, Kevin Ratnasekera <[email protected]> > wrote: > >> Hi Wishmitha, >> >> There are many java clients available for redis, there are several very >> good clients, see comparison [1]. Any reason using jedis over others? >> >> [1] https://redis.io/clients#java >> >> Regards >> Kevin >> >> >> >> On Fri, Jul 14, 2017 at 11:02 PM, Wishmitha Mendis <[email protected]> >> wrote: >> >>> Hi Indika, >>> >>> I have started a basic implementation to test the concurrency handling, >>> message order and transaction support of Redis. Basic message producers and >>> consumers were implemented and messages are stored in a basic Redis queue. >>> GitHub : https://github.com/Wishmitha/RedisTesting >>> >>> Best Regards, >>> >>> On Thu, Jul 13, 2017 at 6:51 PM, Indika Sampath <[email protected]> >>> wrote: >>> >>>> Hi Wishmitha, >>>> >>>> No SQL database is not an option as we have prior experience with >>>> Cassandra. Basically, the important part is to find out performance and >>>> fault tolerance capability in the database. Message broker >>>> read/write/delete records all the time and database should be able to >>>> handle high concurrency. Some metadata and message content write as byte >>>> data. Also, message order matters when delivery. Hence database should >>>> capable to sort records rather than bringing that logic into broker code. >>>> As you mentioned transaction is another factor we are looking in the >>>> database. Shall we start POC with Redis and see how it goes? >>>> >>>> Cheers! >>>> >>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I am working on a project to replace the current RDBMS based database >>>>> of the message broker store with a file based database system. I have been >>>>> researching on potential file based databases which can be used in the >>>>> project scenario. I have evaluated the pros and cons of the each of the >>>>> potential databases mainly based on read and write performances, >>>>> transaction support, data structure (queue) implementation support and >>>>> overhead of replacement. >>>>> >>>>> A summary of database evaluation is represented in the following table. >>>>> >>>>> >>>>> Database >>>>> >>>>> Pros >>>>> >>>>> Cons >>>>> >>>>> LevelDB >>>>> >>>>> - >>>>> >>>>> A simple key value storage >>>>> - >>>>> >>>>> Used in ActiveMQ (replaced with KahaDB) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast due to sorted keys >>>>> 2. >>>>> >>>>> Simple transaction support with batch( ) operation >>>>> 3. >>>>> >>>>> Sublevel feature (provides more organization and transactions >>>>> facilitation) >>>>> 4. >>>>> >>>>> LiveStream feature (able to query things which are yet being >>>>> receiving) >>>>> 5. >>>>> >>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>> >>>>> >>>>> 1. >>>>> >>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>> (improved in RocksDB.) >>>>> 2. >>>>> >>>>> No object relational mapping. >>>>> 3. >>>>> >>>>> Poor performance when memory is not enough. >>>>> >>>>> MongoDB >>>>> >>>>> >>>>> - >>>>> >>>>> Document oriented >>>>> - >>>>> >>>>> Can define JSON schemas >>>>> >>>>> >>>>> >>>>> 1. >>>>> >>>>> Direct object mapping (good replacement for RDBMS) >>>>> 2. >>>>> >>>>> Can implement queue like data structures (store messages in >>>>> separate documents and delete them once delivered / already implemented >>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>> 3. >>>>> >>>>> Capped collections (automatically remove old docs to make space >>>>> for new docs.) >>>>> >>>>> >>>>> 1. >>>>> >>>>> No transaction support for multiple documents. >>>>> 2. >>>>> >>>>> Low performance compared to other NoSQL databases. >>>>> >>>>> Caassandra >>>>> >>>>> >>>>> - >>>>> >>>>> Column oriented (a key is denoted by row and an attribute is >>>>> denoted by a column which can dynamically change.) >>>>> >>>>> >>>>> - >>>>> >>>>> Node - Cluster configuration (all nodes are identical) >>>>> >>>>> >>>>> 1. >>>>> >>>>> High availability (No single point of failure. In case of node >>>>> failure other identical nodes are available.) >>>>> 2. >>>>> >>>>> No data loss due to replication (identical nodes) >>>>> 3. >>>>> >>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>> 4. >>>>> >>>>> Similar query language to SQL (CQL) >>>>> 5. >>>>> >>>>> High performance (No master node concept. Any node can provide >>>>> requested query. No performance bottleneck.) >>>>> 6. >>>>> >>>>> Transaction implementation >>>>> >>>>> >>>>> 1. >>>>> >>>>> Does not support queue structure (message queue implementation is >>>>> an anti-pattern : http://www.datastax.com/dev/bl >>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>> >>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>> ) >>>>> >>>>> Redis >>>>> >>>>> >>>>> - >>>>> >>>>> A simple key value storage >>>>> - >>>>> >>>>> In memory database >>>>> - >>>>> >>>>> Support data structures like lists, caped lists, hashes, sets etc. >>>>> (https://redis.io/topics/data-types-intro) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast read,write performances >>>>> 2. >>>>> >>>>> Provide persistence by writing to hard disk in configurable >>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>> 3. >>>>> >>>>> Can implement message queue structures. ( >>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>> e-queue-with-redis/ >>>>> >>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>> ) >>>>> 4. >>>>> >>>>> Use to implement message stores. (https://redis.io/topics/pubsub) >>>>> 5. >>>>> >>>>> In built basic transaction support (MULTI/EXEC commands : >>>>> https://redis.io/topics/transactions >>>>> <https://redis.io/topics/transactions>) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Loss of data (in case of power outage etc.) >>>>> 2. >>>>> >>>>> Depends on CPU performances when data amount is too high ( >>>>> https://redis.io/topics/persistence >>>>> <https://redis.io/topics/persistence>) >>>>> >>>>> RocksDB >>>>> >>>>> >>>>> - >>>>> >>>>> A simple key value storage (upgrade of LevelDB) >>>>> - >>>>> >>>>> In memory database >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast read,write performances (faster than LevelDB) >>>>> 2. >>>>> >>>>> Can implement queue structures (https://github.com/facebook/r >>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>> >>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>> ) >>>>> 3. >>>>> >>>>> Support concurrency >>>>> 4. >>>>> >>>>> Highly configurable (Pluggable Architecture) >>>>> 5. >>>>> >>>>> Support persistence (memtables and transactional logs manage in >>>>> memory. Memtable is flushed to a sstfile to provide persistence.) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Loss of data (in case of power outage etc.) >>>>> 2. >>>>> >>>>> No in built transaction support (have to use TansactionDB : >>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>> >>>>> >>>>> According to this evaluation I suggest Redis as the most suitable >>>>> database for the message broker store. Even though it has an element of >>>>> risk in data loss, the persistence storing is configurable unlike in other >>>>> key-value stores. And it provides fast read and write performances >>>>> compared >>>>> to other databases with basic transaction support. >>>>> >>>>> And this is basically my evaluation and the opinion. It would be great >>>>> to have a feedback on this, specially regarding the additional criteria to >>>>> consider and other suggested databases. >>>>> >>>>> Best Regards >>>>> >>>>> -- >>>>> >>>>> *Wishmitha Mendis* >>>>> >>>>> *Intern - Software Engineering* >>>>> *WSO2* >>>>> >>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Indika Sampath >>>> Associate Technical Lead >>>> WSO2 Inc. >>>> http://wso2.com >>>> >>>> Phone: +94 716 424 744 <+94%2071%20642%204744> >>>> Blog: http://indikasampath.blogspot.com/ >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> *Wishmitha Mendis* >>> >>> *Intern - Software Engineering* >>> *WSO2* >>> >>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Kevin Ratnaskera* >> Software Engineer >> WSO2 Inc. - http://wso2.com >> lean . enterprise . middleware >> Mobile - +94774696950 <+94%2077%20469%206950> >> Linkedin - https://www.linkedin.com/in/djkevincr >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Wishmitha Mendis* > > *Intern - Software Engineering* > *WSO2* > > *Mobile : +94 777577706* > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Manoj Gunawardena Tech Lead WSO2, Inc.: http://wso2.com lean.enterprise.middleware Mobile : +94 77 2291643
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
