Hi Wishmitha, No SQL database is not an option as we have prior experience with Cassandra. Basically, the important part is to find out performance and fault tolerance capability in the database. Message broker read/write/delete records all the time and database should be able to handle high concurrency. Some metadata and message content write as byte data. Also, message order matters when delivery. Hence database should capable to sort records rather than bringing that logic into broker code. As you mentioned transaction is another factor we are looking in the database. Shall we start POC with Redis and see how it goes?
Cheers! On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> wrote: > Hi all, > > I am working on a project to replace the current RDBMS based database of > the message broker store with a file based database system. I have been > researching on potential file based databases which can be used in the > project scenario. I have evaluated the pros and cons of the each of the > potential databases mainly based on read and write performances, > transaction support, data structure (queue) implementation support and > overhead of replacement. > > A summary of database evaluation is represented in the following table. > > > Database > > Pros > > Cons > > LevelDB > > - > > A simple key value storage > - > > Used in ActiveMQ (replaced with KahaDB) > > > 1. > > Fast due to sorted keys > 2. > > Simple transaction support with batch( ) operation > 3. > > Sublevel feature (provides more organization and transactions > facilitation) > 4. > > LiveStream feature (able to query things which are yet being receiving) > 5. > > Support triggers, groupby (mapreduce) and join (fanout) etc. > > > 1. > > Performance decrease in concurrent / multi threaded scenarios. > (improved in RocksDB.) > 2. > > No object relational mapping. > 3. > > Poor performance when memory is not enough. > > MongoDB > > > - > > Document oriented > - > > Can define JSON schemas > > > > 1. > > Direct object mapping (good replacement for RDBMS) > 2. > > Can implement queue like data structures (store messages in separate > documents and delete them once delivered / already implemented data > structures : https://www.npmjs.com/package/mongodb-queue) > 3. > > Capped collections (automatically remove old docs to make space for > new docs.) > > > 1. > > No transaction support for multiple documents. > 2. > > Low performance compared to other NoSQL databases. > > Caassandra > > > - > > Column oriented (a key is denoted by row and an attribute is denoted > by a column which can dynamically change.) > > > - > > Node - Cluster configuration (all nodes are identical) > > > 1. > > High availability (No single point of failure. In case of node failure > other identical nodes are available.) > 2. > > No data loss due to replication (identical nodes) > 3. > > Easy to model (column store is very similar to RDBMS tables.) > 4. > > Similar query language to SQL (CQL) > 5. > > High performance (No master node concept. Any node can provide > requested query. No performance bottleneck.) > 6. > > Transaction implementation > > > 1. > > Does not support queue structure (message queue implementation is an > anti-pattern : http://www.datastax.com/dev/ > blog/cassandra-anti-patterns-queues-and-queue-like-datasets > > <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> > ) > > Redis > > > - > > A simple key value storage > - > > In memory database > - > > Support data structures like lists, caped lists, hashes, sets etc. ( > https://redis.io/topics/data-types-intro > <https://redis.io/topics/data-types-intro>) > > > 1. > > Fast read,write performances > 2. > > Provide persistence by writing to hard disk in configurable intervals. > (snapshots : https://redis.io/topics/persistence) > 3. > > Can implement message queue structures. (http://fiznool.com/blog/2016/ > 02/24/building-a-simple-message-queue-with-redis/ > > <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> > ) > 4. > > Use to implement message stores. (https://redis.io/topics/pubsub) > 5. > > In built basic transaction support (MULTI/EXEC commands : > https://redis.io/topics/transactions > <https://redis.io/topics/transactions>) > > > 1. > > Loss of data (in case of power outage etc.) > 2. > > Depends on CPU performances when data amount is too high ( > https://redis.io/topics/persistence > <https://redis.io/topics/persistence>) > > RocksDB > > > - > > A simple key value storage (upgrade of LevelDB) > - > > In memory database > > > 1. > > Fast read,write performances (faster than LevelDB) > 2. > > Can implement queue structures (https://github.com/facebook/ > rocksdb/wiki/Implement-Queue-Service-Using-RocksDB > > <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> > ) > 3. > > Support concurrency > 4. > > Highly configurable (Pluggable Architecture) > 5. > > Support persistence (memtables and transactional logs manage in > memory. Memtable is flushed to a sstfile to provide persistence.) > > > 1. > > Loss of data (in case of power outage etc.) > 2. > > No in built transaction support (have to use TansactionDB : > https://github.com/facebook/rocksdb/wiki/Transactions > <https://github.com/facebook/rocksdb/wiki/Transactions> ) > > > According to this evaluation I suggest Redis as the most suitable database > for the message broker store. Even though it has an element of risk in data > loss, the persistence storing is configurable unlike in other key-value > stores. And it provides fast read and write performances compared to other > databases with basic transaction support. > > And this is basically my evaluation and the opinion. It would be great to > have a feedback on this, specially regarding the additional criteria to > consider and other suggested databases. > > Best Regards > > -- > > *Wishmitha Mendis* > > *Intern - Software Engineering* > *WSO2* > > *Mobile : +94 777577706 <+94%2077%20757%207706>* > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Indika Sampath Associate Technical Lead WSO2 Inc. http://wso2.com Phone: +94 716 424 744 Blog: http://indikasampath.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
