Hi Wishmitha, I have done some research work about existing message brokers previously, and here I have attach the link to the document which I have recorded the details about ActiveMQ Artemis high performance file journal and other file based message broker systems [1]. I think it will be useful to get an idea about how and why existing message brokers have implemented file based message stores.
[1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing Thanks, On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> wrote: > Hi all, > > I am working on a project to replace the current RDBMS based database of > the message broker store with a file based database system. I have been > researching on potential file based databases which can be used in the > project scenario. I have evaluated the pros and cons of the each of the > potential databases mainly based on read and write performances, > transaction support, data structure (queue) implementation support and > overhead of replacement. > > A summary of database evaluation is represented in the following table. > > > Database > > Pros > > Cons > > LevelDB > > - > > A simple key value storage > - > > Used in ActiveMQ (replaced with KahaDB) > > > 1. > > Fast due to sorted keys > 2. > > Simple transaction support with batch( ) operation > 3. > > Sublevel feature (provides more organization and transactions > facilitation) > 4. > > LiveStream feature (able to query things which are yet being receiving) > 5. > > Support triggers, groupby (mapreduce) and join (fanout) etc. > > > 1. > > Performance decrease in concurrent / multi threaded scenarios. > (improved in RocksDB.) > 2. > > No object relational mapping. > 3. > > Poor performance when memory is not enough. > > MongoDB > > > - > > Document oriented > - > > Can define JSON schemas > > > > 1. > > Direct object mapping (good replacement for RDBMS) > 2. > > Can implement queue like data structures (store messages in separate > documents and delete them once delivered / already implemented data > structures : https://www.npmjs.com/package/mongodb-queue) > 3. > > Capped collections (automatically remove old docs to make space for > new docs.) > > > 1. > > No transaction support for multiple documents. > 2. > > Low performance compared to other NoSQL databases. > > Caassandra > > > - > > Column oriented (a key is denoted by row and an attribute is denoted > by a column which can dynamically change.) > > > - > > Node - Cluster configuration (all nodes are identical) > > > 1. > > High availability (No single point of failure. In case of node failure > other identical nodes are available.) > 2. > > No data loss due to replication (identical nodes) > 3. > > Easy to model (column store is very similar to RDBMS tables.) > 4. > > Similar query language to SQL (CQL) > 5. > > High performance (No master node concept. Any node can provide > requested query. No performance bottleneck.) > 6. > > Transaction implementation > > > 1. > > Does not support queue structure (message queue implementation is an > anti-pattern : http://www.datastax.com/dev/ > blog/cassandra-anti-patterns-queues-and-queue-like-datasets > > <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> > ) > > Redis > > > - > > A simple key value storage > - > > In memory database > - > > Support data structures like lists, caped lists, hashes, sets etc. ( > https://redis.io/topics/data-types-intro > <https://redis.io/topics/data-types-intro>) > > > 1. > > Fast read,write performances > 2. > > Provide persistence by writing to hard disk in configurable intervals. > (snapshots : https://redis.io/topics/persistence) > 3. > > Can implement message queue structures. (http://fiznool.com/blog/2016/ > 02/24/building-a-simple-message-queue-with-redis/ > > <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> > ) > 4. > > Use to implement message stores. (https://redis.io/topics/pubsub) > 5. > > In built basic transaction support (MULTI/EXEC commands : > https://redis.io/topics/transactions > <https://redis.io/topics/transactions>) > > > 1. > > Loss of data (in case of power outage etc.) > 2. > > Depends on CPU performances when data amount is too high ( > https://redis.io/topics/persistence > <https://redis.io/topics/persistence>) > > RocksDB > > > - > > A simple key value storage (upgrade of LevelDB) > - > > In memory database > > > 1. > > Fast read,write performances (faster than LevelDB) > 2. > > Can implement queue structures (https://github.com/facebook/ > rocksdb/wiki/Implement-Queue-Service-Using-RocksDB > > <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> > ) > 3. > > Support concurrency > 4. > > Highly configurable (Pluggable Architecture) > 5. > > Support persistence (memtables and transactional logs manage in > memory. Memtable is flushed to a sstfile to provide persistence.) > > > 1. > > Loss of data (in case of power outage etc.) > 2. > > No in built transaction support (have to use TansactionDB : > https://github.com/facebook/rocksdb/wiki/Transactions > <https://github.com/facebook/rocksdb/wiki/Transactions> ) > > > According to this evaluation I suggest Redis as the most suitable database > for the message broker store. Even though it has an element of risk in data > loss, the persistence storing is configurable unlike in other key-value > stores. And it provides fast read and write performances compared to other > databases with basic transaction support. > > And this is basically my evaluation and the opinion. It would be great to > have a feedback on this, specially regarding the additional criteria to > consider and other suggested databases. > > Best Regards > > -- > > *Wishmitha Mendis* > > *Intern - Software Engineering* > *WSO2* > > *Mobile : +94 777577706 <+94%2077%20757%207706>* > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malaka. -- Malaka Gangananda - Software Engineer | WSO2 Email : [email protected] Mobile : +94713564340 Web : http://wso2.com <http://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
