Hi Wishmitha, If we use AOF wth fsync at every query, will there be a data loss with Redis? AOF might not have very good performance compared to RDB. But as Sasikal mentioned we cannot allow message losses in the broker.
On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[email protected]> wrote: > Dear Wishmitha, > > For all the databases mentioned please provide references to the relevant > websites and where you got the information from. You have mentioned for > some which is great and you should do so for others as well. Also when > referencing don't have the weblink inline with the statement because it > distracts the viewer as the links might be long. You can number the > references and refer them below (You don't need to follow any standards, > simply giving the link would suffice). > > Cheers, > Riyafa > > On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[email protected]> > wrote: > >> Hi Wishmitha, >> >> We cannot tolerate data loss in the system. Guaranteed delivery comes >> first, performance comes next for the broker. >> >> Thank you, >> Sasikala >> >> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> >> wrote: >> >>> Hi All, >>> >>> I have attached the pdf of above mentioned document. >>> >>> Thanks. >>> >>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected]> >>> wrote: >>> >>>> Hi Wishmitha, >>>> >>>> I have done some research work about existing message brokers >>>> previously, and here I have attach the link to the document which I have >>>> recorded the details about ActiveMQ Artemis high performance file journal >>>> and other file based message broker systems [1]. I think it will be useful >>>> to get an idea about how and why existing message brokers have implemented >>>> file based message stores. >>>> >>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >>>> >>>> Thanks, >>>> >>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I am working on a project to replace the current RDBMS based database >>>>> of the message broker store with a file based database system. I have been >>>>> researching on potential file based databases which can be used in the >>>>> project scenario. I have evaluated the pros and cons of the each of the >>>>> potential databases mainly based on read and write performances, >>>>> transaction support, data structure (queue) implementation support and >>>>> overhead of replacement. >>>>> >>>>> A summary of database evaluation is represented in the following table. >>>>> >>>>> >>>>> Database >>>>> >>>>> Pros >>>>> >>>>> Cons >>>>> >>>>> LevelDB >>>>> >>>>> - >>>>> >>>>> A simple key value storage >>>>> - >>>>> >>>>> Used in ActiveMQ (replaced with KahaDB) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast due to sorted keys >>>>> 2. >>>>> >>>>> Simple transaction support with batch( ) operation >>>>> 3. >>>>> >>>>> Sublevel feature (provides more organization and transactions >>>>> facilitation) >>>>> 4. >>>>> >>>>> LiveStream feature (able to query things which are yet being >>>>> receiving) >>>>> 5. >>>>> >>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>> >>>>> >>>>> 1. >>>>> >>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>> (improved in RocksDB.) >>>>> 2. >>>>> >>>>> No object relational mapping. >>>>> 3. >>>>> >>>>> Poor performance when memory is not enough. >>>>> >>>>> MongoDB >>>>> >>>>> >>>>> - >>>>> >>>>> Document oriented >>>>> - >>>>> >>>>> Can define JSON schemas >>>>> >>>>> >>>>> >>>>> 1. >>>>> >>>>> Direct object mapping (good replacement for RDBMS) >>>>> 2. >>>>> >>>>> Can implement queue like data structures (store messages in >>>>> separate documents and delete them once delivered / already implemented >>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>> 3. >>>>> >>>>> Capped collections (automatically remove old docs to make space >>>>> for new docs.) >>>>> >>>>> >>>>> 1. >>>>> >>>>> No transaction support for multiple documents. >>>>> 2. >>>>> >>>>> Low performance compared to other NoSQL databases. >>>>> >>>>> Caassandra >>>>> >>>>> >>>>> - >>>>> >>>>> Column oriented (a key is denoted by row and an attribute is >>>>> denoted by a column which can dynamically change.) >>>>> >>>>> >>>>> - >>>>> >>>>> Node - Cluster configuration (all nodes are identical) >>>>> >>>>> >>>>> 1. >>>>> >>>>> High availability (No single point of failure. In case of node >>>>> failure other identical nodes are available.) >>>>> 2. >>>>> >>>>> No data loss due to replication (identical nodes) >>>>> 3. >>>>> >>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>> 4. >>>>> >>>>> Similar query language to SQL (CQL) >>>>> 5. >>>>> >>>>> High performance (No master node concept. Any node can provide >>>>> requested query. No performance bottleneck.) >>>>> 6. >>>>> >>>>> Transaction implementation >>>>> >>>>> >>>>> 1. >>>>> >>>>> Does not support queue structure (message queue implementation is >>>>> an anti-pattern : http://www.datastax.com/dev/bl >>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>> >>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>> ) >>>>> >>>>> Redis >>>>> >>>>> >>>>> - >>>>> >>>>> A simple key value storage >>>>> - >>>>> >>>>> In memory database >>>>> - >>>>> >>>>> Support data structures like lists, caped lists, hashes, sets etc. >>>>> (https://redis.io/topics/data-types-intro) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast read,write performances >>>>> 2. >>>>> >>>>> Provide persistence by writing to hard disk in configurable >>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>> 3. >>>>> >>>>> Can implement message queue structures. ( >>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>> e-queue-with-redis/ >>>>> >>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>> ) >>>>> 4. >>>>> >>>>> Use to implement message stores. (https://redis.io/topics/pubsub) >>>>> 5. >>>>> >>>>> In built basic transaction support (MULTI/EXEC commands : >>>>> https://redis.io/topics/transactions >>>>> <https://redis.io/topics/transactions>) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Loss of data (in case of power outage etc.) >>>>> 2. >>>>> >>>>> Depends on CPU performances when data amount is too high ( >>>>> https://redis.io/topics/persistence >>>>> <https://redis.io/topics/persistence>) >>>>> >>>>> RocksDB >>>>> >>>>> >>>>> - >>>>> >>>>> A simple key value storage (upgrade of LevelDB) >>>>> - >>>>> >>>>> In memory database >>>>> >>>>> >>>>> 1. >>>>> >>>>> Fast read,write performances (faster than LevelDB) >>>>> 2. >>>>> >>>>> Can implement queue structures (https://github.com/facebook/r >>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>> >>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>> ) >>>>> 3. >>>>> >>>>> Support concurrency >>>>> 4. >>>>> >>>>> Highly configurable (Pluggable Architecture) >>>>> 5. >>>>> >>>>> Support persistence (memtables and transactional logs manage in >>>>> memory. Memtable is flushed to a sstfile to provide persistence.) >>>>> >>>>> >>>>> 1. >>>>> >>>>> Loss of data (in case of power outage etc.) >>>>> 2. >>>>> >>>>> No in built transaction support (have to use TansactionDB : >>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>> >>>>> >>>>> According to this evaluation I suggest Redis as the most suitable >>>>> database for the message broker store. Even though it has an element of >>>>> risk in data loss, the persistence storing is configurable unlike in other >>>>> key-value stores. And it provides fast read and write performances >>>>> compared >>>>> to other databases with basic transaction support. >>>>> >>>>> And this is basically my evaluation and the opinion. It would be great >>>>> to have a feedback on this, specially regarding the additional criteria to >>>>> consider and other suggested databases. >>>>> >>>>> Best Regards >>>>> >>>>> -- >>>>> >>>>> *Wishmitha Mendis* >>>>> >>>>> *Intern - Software Engineering* >>>>> *WSO2* >>>>> >>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Malaka. >>>> -- >>>> Malaka Gangananda - Software Engineer | WSO2 >>>> Email : [email protected] >>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>> Web : http://wso2.com >>>> <http://wso2.com/signature> >>>> >>> >>> >>> >>> -- >>> Malaka. >>> -- >>> Malaka Gangananda - Software Engineer | WSO2 >>> Email : [email protected] >>> Mobile : +94713564340 <+94%2071%20356%204340> >>> Web : http://wso2.com >>> <http://wso2.com/signature> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Sasikala Kottegoda >> *Senior Software Engineer* >> WSO2 Inc., http://wso2.com/ >> lean. enterprise. middleware >> Mobile: +94 774835928 <+94%2077%20483%205928> >> >> [image: https://wso2.com/signature] <https://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Riyafa Abdul Hameed > Software Engineer, WSO2 Lanka (Pvt) Ltd <http://wso2.com/> > > Email: [email protected] <[email protected]> > Website: https://riyafa.wordpress.com/ <http://riyafa.wordpress.com/> > <http://facebook.com/riyafa.ahf> <http://lk.linkedin.com/in/riyafa> > <http://twitter.com/Riyafa1> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Asanka Abeyweera Senior Software Engineer WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
