Hi Wishmitha, We cannot tolerate data loss in the system. Guaranteed delivery comes first, performance comes next for the broker.
Thank you, Sasikala On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> wrote: > Hi All, > > I have attached the pdf of above mentioned document. > > Thanks. > > On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected]> > wrote: > >> Hi Wishmitha, >> >> I have done some research work about existing message brokers previously, >> and here I have attach the link to the document which I have recorded the >> details about ActiveMQ Artemis high performance file journal and other file >> based message broker systems [1]. I think it will be useful to get an idea >> about how and why existing message brokers have implemented file based >> message stores. >> >> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >> >> Thanks, >> >> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> >> wrote: >> >>> Hi all, >>> >>> I am working on a project to replace the current RDBMS based database of >>> the message broker store with a file based database system. I have been >>> researching on potential file based databases which can be used in the >>> project scenario. I have evaluated the pros and cons of the each of the >>> potential databases mainly based on read and write performances, >>> transaction support, data structure (queue) implementation support and >>> overhead of replacement. >>> >>> A summary of database evaluation is represented in the following table. >>> >>> >>> Database >>> >>> Pros >>> >>> Cons >>> >>> LevelDB >>> >>> - >>> >>> A simple key value storage >>> - >>> >>> Used in ActiveMQ (replaced with KahaDB) >>> >>> >>> 1. >>> >>> Fast due to sorted keys >>> 2. >>> >>> Simple transaction support with batch( ) operation >>> 3. >>> >>> Sublevel feature (provides more organization and transactions >>> facilitation) >>> 4. >>> >>> LiveStream feature (able to query things which are yet being >>> receiving) >>> 5. >>> >>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>> >>> >>> 1. >>> >>> Performance decrease in concurrent / multi threaded scenarios. >>> (improved in RocksDB.) >>> 2. >>> >>> No object relational mapping. >>> 3. >>> >>> Poor performance when memory is not enough. >>> >>> MongoDB >>> >>> >>> - >>> >>> Document oriented >>> - >>> >>> Can define JSON schemas >>> >>> >>> >>> 1. >>> >>> Direct object mapping (good replacement for RDBMS) >>> 2. >>> >>> Can implement queue like data structures (store messages in separate >>> documents and delete them once delivered / already implemented data >>> structures : https://www.npmjs.com/package/mongodb-queue) >>> 3. >>> >>> Capped collections (automatically remove old docs to make space for >>> new docs.) >>> >>> >>> 1. >>> >>> No transaction support for multiple documents. >>> 2. >>> >>> Low performance compared to other NoSQL databases. >>> >>> Caassandra >>> >>> >>> - >>> >>> Column oriented (a key is denoted by row and an attribute is denoted >>> by a column which can dynamically change.) >>> >>> >>> - >>> >>> Node - Cluster configuration (all nodes are identical) >>> >>> >>> 1. >>> >>> High availability (No single point of failure. In case of node >>> failure other identical nodes are available.) >>> 2. >>> >>> No data loss due to replication (identical nodes) >>> 3. >>> >>> Easy to model (column store is very similar to RDBMS tables.) >>> 4. >>> >>> Similar query language to SQL (CQL) >>> 5. >>> >>> High performance (No master node concept. Any node can provide >>> requested query. No performance bottleneck.) >>> 6. >>> >>> Transaction implementation >>> >>> >>> 1. >>> >>> Does not support queue structure (message queue implementation is an >>> anti-pattern : http://www.datastax.com/dev/bl >>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>> >>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>> ) >>> >>> Redis >>> >>> >>> - >>> >>> A simple key value storage >>> - >>> >>> In memory database >>> - >>> >>> Support data structures like lists, caped lists, hashes, sets etc. ( >>> https://redis.io/topics/data-types-intro >>> <https://redis.io/topics/data-types-intro>) >>> >>> >>> 1. >>> >>> Fast read,write performances >>> 2. >>> >>> Provide persistence by writing to hard disk in configurable >>> intervals. (snapshots : https://redis.io/topics/persistence) >>> 3. >>> >>> Can implement message queue structures. ( >>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>> e-queue-with-redis/ >>> >>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>> ) >>> 4. >>> >>> Use to implement message stores. (https://redis.io/topics/pubsub) >>> 5. >>> >>> In built basic transaction support (MULTI/EXEC commands : >>> https://redis.io/topics/transactions >>> <https://redis.io/topics/transactions>) >>> >>> >>> 1. >>> >>> Loss of data (in case of power outage etc.) >>> 2. >>> >>> Depends on CPU performances when data amount is too high ( >>> https://redis.io/topics/persistence >>> <https://redis.io/topics/persistence>) >>> >>> RocksDB >>> >>> >>> - >>> >>> A simple key value storage (upgrade of LevelDB) >>> - >>> >>> In memory database >>> >>> >>> 1. >>> >>> Fast read,write performances (faster than LevelDB) >>> 2. >>> >>> Can implement queue structures (https://github.com/facebook/r >>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>> >>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>> ) >>> 3. >>> >>> Support concurrency >>> 4. >>> >>> Highly configurable (Pluggable Architecture) >>> 5. >>> >>> Support persistence (memtables and transactional logs manage in >>> memory. Memtable is flushed to a sstfile to provide persistence.) >>> >>> >>> 1. >>> >>> Loss of data (in case of power outage etc.) >>> 2. >>> >>> No in built transaction support (have to use TansactionDB : >>> https://github.com/facebook/rocksdb/wiki/Transactions >>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>> >>> >>> According to this evaluation I suggest Redis as the most suitable >>> database for the message broker store. Even though it has an element of >>> risk in data loss, the persistence storing is configurable unlike in other >>> key-value stores. And it provides fast read and write performances compared >>> to other databases with basic transaction support. >>> >>> And this is basically my evaluation and the opinion. It would be great >>> to have a feedback on this, specially regarding the additional criteria to >>> consider and other suggested databases. >>> >>> Best Regards >>> >>> -- >>> >>> *Wishmitha Mendis* >>> >>> *Intern - Software Engineering* >>> *WSO2* >>> >>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Malaka. >> -- >> Malaka Gangananda - Software Engineer | WSO2 >> Email : [email protected] >> Mobile : +94713564340 <+94%2071%20356%204340> >> Web : http://wso2.com >> <http://wso2.com/signature> >> > > > > -- > Malaka. > -- > Malaka Gangananda - Software Engineer | WSO2 > Email : [email protected] > Mobile : +94713564340 <+94%2071%20356%204340> > Web : http://wso2.com > <http://wso2.com/signature> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sasikala Kottegoda *Senior Software Engineer* WSO2 Inc., http://wso2.com/ lean. enterprise. middleware Mobile: +94 774835928 [image: https://wso2.com/signature] <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
