Hi all, I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. I have been researching on potential file based databases which can be used in the project scenario. I have evaluated the pros and cons of the each of the potential databases mainly based on read and write performances, transaction support, data structure (queue) implementation support and overhead of replacement.
A summary of database evaluation is represented in the following table. Database Pros Cons LevelDB - A simple key value storage - Used in ActiveMQ (replaced with KahaDB) 1. Fast due to sorted keys 2. Simple transaction support with batch( ) operation 3. Sublevel feature (provides more organization and transactions facilitation) 4. LiveStream feature (able to query things which are yet being receiving) 5. Support triggers, groupby (mapreduce) and join (fanout) etc. 1. Performance decrease in concurrent / multi threaded scenarios. (improved in RocksDB.) 2. No object relational mapping. 3. Poor performance when memory is not enough. MongoDB - Document oriented - Can define JSON schemas 1. Direct object mapping (good replacement for RDBMS) 2. Can implement queue like data structures (store messages in separate documents and delete them once delivered / already implemented data structures : https://www.npmjs.com/package/mongodb-queue) 3. Capped collections (automatically remove old docs to make space for new docs.) 1. No transaction support for multiple documents. 2. Low performance compared to other NoSQL databases. Caassandra - Column oriented (a key is denoted by row and an attribute is denoted by a column which can dynamically change.) - Node - Cluster configuration (all nodes are identical) 1. High availability (No single point of failure. In case of node failure other identical nodes are available.) 2. No data loss due to replication (identical nodes) 3. Easy to model (column store is very similar to RDBMS tables.) 4. Similar query language to SQL (CQL) 5. High performance (No master node concept. Any node can provide requested query. No performance bottleneck.) 6. Transaction implementation 1. Does not support queue structure (message queue implementation is an anti-pattern : http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets ) Redis - A simple key value storage - In memory database - Support data structures like lists, caped lists, hashes, sets etc. ( https://redis.io/topics/data-types-intro) 1. Fast read,write performances 2. Provide persistence by writing to hard disk in configurable intervals. (snapshots : https://redis.io/topics/persistence) 3. Can implement message queue structures. ( http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/ ) 4. Use to implement message stores. (https://redis.io/topics/pubsub) 5. In built basic transaction support (MULTI/EXEC commands : https://redis.io/topics/transactions) 1. Loss of data (in case of power outage etc.) 2. Depends on CPU performances when data amount is too high ( https://redis.io/topics/persistence) RocksDB - A simple key value storage (upgrade of LevelDB) - In memory database 1. Fast read,write performances (faster than LevelDB) 2. Can implement queue structures ( https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB ) 3. Support concurrency 4. Highly configurable (Pluggable Architecture) 5. Support persistence (memtables and transactional logs manage in memory. Memtable is flushed to a sstfile to provide persistence.) 1. Loss of data (in case of power outage etc.) 2. No in built transaction support (have to use TansactionDB : https://github.com/facebook/rocksdb/wiki/Transactions ) According to this evaluation I suggest Redis as the most suitable database for the message broker store. Even though it has an element of risk in data loss, the persistence storing is configurable unlike in other key-value stores. And it provides fast read and write performances compared to other databases with basic transaction support. And this is basically my evaluation and the opinion. It would be great to have a feedback on this, specially regarding the additional criteria to consider and other suggested databases. Best Regards -- *Wishmitha Mendis* *Intern - Software Engineering* *WSO2* *Mobile : +94 777577706*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
