Hi, >From the given description, it seems persistence is supported by writing to hard disk in configurable intervals. Unless we have a very low interval (which might be against any expected performance improvement brought by the database being in-memory) it cannot be assured that there won't be data loss. Also even a low interval would not fully guarantee that. I had a quick look at [1] and it says "You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.". So, if server crashes we could lose 1 - M changes. We can only reduce it but cannot avoid it completely. :) So I'm not sure if we could use Redis.
[1] https://redis.io/topics/persistence Thanks, Manuri On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[email protected]> wrote: > Hi Wishmitha, > > If we use AOF wth fsync at every query, will there be a data loss with > Redis? AOF might not have very good performance compared to RDB. But as > Sasikal mentioned we cannot allow message losses in the broker. > > On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[email protected]> > wrote: > >> Dear Wishmitha, >> >> For all the databases mentioned please provide references to the relevant >> websites and where you got the information from. You have mentioned for >> some which is great and you should do so for others as well. Also when >> referencing don't have the weblink inline with the statement because it >> distracts the viewer as the links might be long. You can number the >> references and refer them below (You don't need to follow any standards, >> simply giving the link would suffice). >> >> Cheers, >> Riyafa >> >> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[email protected]> >> wrote: >> >>> Hi Wishmitha, >>> >>> We cannot tolerate data loss in the system. Guaranteed delivery comes >>> first, performance comes next for the broker. >>> >>> Thank you, >>> Sasikala >>> >>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> I have attached the pdf of above mentioned document. >>>> >>>> Thanks. >>>> >>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected]> >>>> wrote: >>>> >>>>> Hi Wishmitha, >>>>> >>>>> I have done some research work about existing message brokers >>>>> previously, and here I have attach the link to the document which I have >>>>> recorded the details about ActiveMQ Artemis high performance file journal >>>>> and other file based message broker systems [1]. I think it will be useful >>>>> to get an idea about how and why existing message brokers have implemented >>>>> file based message stores. >>>>> >>>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >>>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >>>>> >>>>> Thanks, >>>>> >>>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I am working on a project to replace the current RDBMS based database >>>>>> of the message broker store with a file based database system. I have >>>>>> been >>>>>> researching on potential file based databases which can be used in the >>>>>> project scenario. I have evaluated the pros and cons of the each of the >>>>>> potential databases mainly based on read and write performances, >>>>>> transaction support, data structure (queue) implementation support and >>>>>> overhead of replacement. >>>>>> >>>>>> A summary of database evaluation is represented in the following >>>>>> table. >>>>>> >>>>>> >>>>>> Database >>>>>> >>>>>> Pros >>>>>> >>>>>> Cons >>>>>> >>>>>> LevelDB >>>>>> >>>>>> - >>>>>> >>>>>> A simple key value storage >>>>>> - >>>>>> >>>>>> Used in ActiveMQ (replaced with KahaDB) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Fast due to sorted keys >>>>>> 2. >>>>>> >>>>>> Simple transaction support with batch( ) operation >>>>>> 3. >>>>>> >>>>>> Sublevel feature (provides more organization and transactions >>>>>> facilitation) >>>>>> 4. >>>>>> >>>>>> LiveStream feature (able to query things which are yet being >>>>>> receiving) >>>>>> 5. >>>>>> >>>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>>> (improved in RocksDB.) >>>>>> 2. >>>>>> >>>>>> No object relational mapping. >>>>>> 3. >>>>>> >>>>>> Poor performance when memory is not enough. >>>>>> >>>>>> MongoDB >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Document oriented >>>>>> - >>>>>> >>>>>> Can define JSON schemas >>>>>> >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Direct object mapping (good replacement for RDBMS) >>>>>> 2. >>>>>> >>>>>> Can implement queue like data structures (store messages in >>>>>> separate documents and delete them once delivered / already >>>>>> implemented >>>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>>> 3. >>>>>> >>>>>> Capped collections (automatically remove old docs to make space >>>>>> for new docs.) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> No transaction support for multiple documents. >>>>>> 2. >>>>>> >>>>>> Low performance compared to other NoSQL databases. >>>>>> >>>>>> Caassandra >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Column oriented (a key is denoted by row and an attribute is >>>>>> denoted by a column which can dynamically change.) >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> Node - Cluster configuration (all nodes are identical) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> High availability (No single point of failure. In case of node >>>>>> failure other identical nodes are available.) >>>>>> 2. >>>>>> >>>>>> No data loss due to replication (identical nodes) >>>>>> 3. >>>>>> >>>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>>> 4. >>>>>> >>>>>> Similar query language to SQL (CQL) >>>>>> 5. >>>>>> >>>>>> High performance (No master node concept. Any node can provide >>>>>> requested query. No performance bottleneck.) >>>>>> 6. >>>>>> >>>>>> Transaction implementation >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Does not support queue structure (message queue implementation is >>>>>> an anti-pattern : http://www.datastax.com/dev/bl >>>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>>> >>>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>>> ) >>>>>> >>>>>> Redis >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> A simple key value storage >>>>>> - >>>>>> >>>>>> In memory database >>>>>> - >>>>>> >>>>>> Support data structures like lists, caped lists, hashes, sets >>>>>> etc. (https://redis.io/topics/data-types-intro) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Fast read,write performances >>>>>> 2. >>>>>> >>>>>> Provide persistence by writing to hard disk in configurable >>>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>>> 3. >>>>>> >>>>>> Can implement message queue structures. ( >>>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>>> e-queue-with-redis/ >>>>>> >>>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>>> ) >>>>>> 4. >>>>>> >>>>>> Use to implement message stores. (https://redis.io/topics/pubsub) >>>>>> 5. >>>>>> >>>>>> In built basic transaction support (MULTI/EXEC commands : >>>>>> https://redis.io/topics/transactions >>>>>> <https://redis.io/topics/transactions>) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Loss of data (in case of power outage etc.) >>>>>> 2. >>>>>> >>>>>> Depends on CPU performances when data amount is too high ( >>>>>> https://redis.io/topics/persistence >>>>>> <https://redis.io/topics/persistence>) >>>>>> >>>>>> RocksDB >>>>>> >>>>>> >>>>>> - >>>>>> >>>>>> A simple key value storage (upgrade of LevelDB) >>>>>> - >>>>>> >>>>>> In memory database >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Fast read,write performances (faster than LevelDB) >>>>>> 2. >>>>>> >>>>>> Can implement queue structures (https://github.com/facebook/r >>>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>>> >>>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>>> ) >>>>>> 3. >>>>>> >>>>>> Support concurrency >>>>>> 4. >>>>>> >>>>>> Highly configurable (Pluggable Architecture) >>>>>> 5. >>>>>> >>>>>> Support persistence (memtables and transactional logs manage in >>>>>> memory. Memtable is flushed to a sstfile to provide persistence.) >>>>>> >>>>>> >>>>>> 1. >>>>>> >>>>>> Loss of data (in case of power outage etc.) >>>>>> 2. >>>>>> >>>>>> No in built transaction support (have to use TansactionDB : >>>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>>> >>>>>> >>>>>> According to this evaluation I suggest Redis as the most suitable >>>>>> database for the message broker store. Even though it has an element of >>>>>> risk in data loss, the persistence storing is configurable unlike in >>>>>> other >>>>>> key-value stores. And it provides fast read and write performances >>>>>> compared >>>>>> to other databases with basic transaction support. >>>>>> >>>>>> And this is basically my evaluation and the opinion. It would be >>>>>> great to have a feedback on this, specially regarding the additional >>>>>> criteria to consider and other suggested databases. >>>>>> >>>>>> Best Regards >>>>>> >>>>>> -- >>>>>> >>>>>> *Wishmitha Mendis* >>>>>> >>>>>> *Intern - Software Engineering* >>>>>> *WSO2* >>>>>> >>>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Malaka. >>>>> -- >>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>> Email : [email protected] >>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>> Web : http://wso2.com >>>>> <http://wso2.com/signature> >>>>> >>>> >>>> >>>> >>>> -- >>>> Malaka. >>>> -- >>>> Malaka Gangananda - Software Engineer | WSO2 >>>> Email : [email protected] >>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>> Web : http://wso2.com >>>> <http://wso2.com/signature> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Sasikala Kottegoda >>> *Senior Software Engineer* >>> WSO2 Inc., http://wso2.com/ >>> lean. enterprise. middleware >>> Mobile: +94 774835928 <+94%2077%20483%205928> >>> >>> [image: https://wso2.com/signature] <https://wso2.com/signature> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Riyafa Abdul Hameed >> Software Engineer, WSO2 Lanka (Pvt) Ltd <http://wso2.com/> >> >> Email: [email protected] <[email protected]> >> Website: https://riyafa.wordpress.com/ <http://riyafa.wordpress.com/> >> <http://facebook.com/riyafa.ahf> <http://lk.linkedin.com/in/riyafa> >> <http://twitter.com/Riyafa1> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Asanka Abeyweera > Senior Software Engineer > WSO2 Inc. > > Phone: +94 712228648 <+94%2071%20222%208648> > Blog: a5anka.github.io > > <https://wso2.com/signature> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Manuri Amaya Perera* *Software Engineer* *WSO2 Inc.* *Blog: http://manuriamayaperera.blogspot.com <http://manuriamayaperera.blogspot.com>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
