Hi Maryam, Even the recommended "appendfsync everysec" mode has a potential data loss, the only option would be trying out fsync at every query. So it would be useful to implement a POC as you suggested based on "appendfsync always" and test the performances whether it can deliver the desired results.
Best Regards, On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[email protected]> wrote: > Hi, > > From the given description, it seems persistence is supported by writing > to hard disk in configurable intervals. Unless we have a very low interval > (which might be against any expected performance improvement brought by the > database being in-memory) it cannot be assured that there won't be data > loss. Also even a low interval would not fully guarantee that. > I had a quick look at [1] and it says "You can configure Redis to have it > save the dataset every N seconds if there are at least M changes in the > dataset, or you can manually call the SAVE or BGSAVE commands.". So, if > server crashes we could lose 1 - M changes. We can only reduce it but > cannot avoid it completely. :) > So I'm not sure if we could use Redis. > > [1] https://redis.io/topics/persistence > > Thanks, > Manuri > > On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[email protected]> > wrote: > >> Hi Wishmitha, >> >> If we use AOF wth fsync at every query, will there be a data loss with >> Redis? AOF might not have very good performance compared to RDB. But as >> Sasikal mentioned we cannot allow message losses in the broker. >> >> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[email protected]> >> wrote: >> >>> Dear Wishmitha, >>> >>> For all the databases mentioned please provide references to the >>> relevant websites and where you got the information from. You have >>> mentioned for some which is great and you should do so for others as well. >>> Also when referencing don't have the weblink inline with the statement >>> because it distracts the viewer as the links might be long. You can number >>> the references and refer them below (You don't need to follow any >>> standards, simply giving the link would suffice). >>> >>> Cheers, >>> Riyafa >>> >>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[email protected]> >>> wrote: >>> >>>> Hi Wishmitha, >>>> >>>> We cannot tolerate data loss in the system. Guaranteed delivery comes >>>> first, performance comes next for the broker. >>>> >>>> Thank you, >>>> Sasikala >>>> >>>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> I have attached the pdf of above mentioned document. >>>>> >>>>> Thanks. >>>>> >>>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Wishmitha, >>>>>> >>>>>> I have done some research work about existing message brokers >>>>>> previously, and here I have attach the link to the document which I have >>>>>> recorded the details about ActiveMQ Artemis high performance file journal >>>>>> and other file based message broker systems [1]. I think it will be >>>>>> useful >>>>>> to get an idea about how and why existing message brokers have >>>>>> implemented >>>>>> file based message stores. >>>>>> >>>>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >>>>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am working on a project to replace the current RDBMS based >>>>>>> database of the message broker store with a file based database system. >>>>>>> I >>>>>>> have been researching on potential file based databases which can be >>>>>>> used >>>>>>> in the project scenario. I have evaluated the pros and cons of the each >>>>>>> of >>>>>>> the potential databases mainly based on read and write performances, >>>>>>> transaction support, data structure (queue) implementation support and >>>>>>> overhead of replacement. >>>>>>> >>>>>>> A summary of database evaluation is represented in the following >>>>>>> table. >>>>>>> >>>>>>> >>>>>>> Database >>>>>>> >>>>>>> Pros >>>>>>> >>>>>>> Cons >>>>>>> >>>>>>> LevelDB >>>>>>> >>>>>>> - >>>>>>> >>>>>>> A simple key value storage >>>>>>> - >>>>>>> >>>>>>> Used in ActiveMQ (replaced with KahaDB) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Fast due to sorted keys >>>>>>> 2. >>>>>>> >>>>>>> Simple transaction support with batch( ) operation >>>>>>> 3. >>>>>>> >>>>>>> Sublevel feature (provides more organization and transactions >>>>>>> facilitation) >>>>>>> 4. >>>>>>> >>>>>>> LiveStream feature (able to query things which are yet being >>>>>>> receiving) >>>>>>> 5. >>>>>>> >>>>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>>>> (improved in RocksDB.) >>>>>>> 2. >>>>>>> >>>>>>> No object relational mapping. >>>>>>> 3. >>>>>>> >>>>>>> Poor performance when memory is not enough. >>>>>>> >>>>>>> MongoDB >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Document oriented >>>>>>> - >>>>>>> >>>>>>> Can define JSON schemas >>>>>>> >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Direct object mapping (good replacement for RDBMS) >>>>>>> 2. >>>>>>> >>>>>>> Can implement queue like data structures (store messages in >>>>>>> separate documents and delete them once delivered / already >>>>>>> implemented >>>>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>>>> 3. >>>>>>> >>>>>>> Capped collections (automatically remove old docs to make space >>>>>>> for new docs.) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> No transaction support for multiple documents. >>>>>>> 2. >>>>>>> >>>>>>> Low performance compared to other NoSQL databases. >>>>>>> >>>>>>> Caassandra >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Column oriented (a key is denoted by row and an attribute is >>>>>>> denoted by a column which can dynamically change.) >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> Node - Cluster configuration (all nodes are identical) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> High availability (No single point of failure. In case of node >>>>>>> failure other identical nodes are available.) >>>>>>> 2. >>>>>>> >>>>>>> No data loss due to replication (identical nodes) >>>>>>> 3. >>>>>>> >>>>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>>>> 4. >>>>>>> >>>>>>> Similar query language to SQL (CQL) >>>>>>> 5. >>>>>>> >>>>>>> High performance (No master node concept. Any node can provide >>>>>>> requested query. No performance bottleneck.) >>>>>>> 6. >>>>>>> >>>>>>> Transaction implementation >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Does not support queue structure (message queue implementation >>>>>>> is an anti-pattern : http://www.datastax.com/dev/bl >>>>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>>>> >>>>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>>>> ) >>>>>>> >>>>>>> Redis >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> A simple key value storage >>>>>>> - >>>>>>> >>>>>>> In memory database >>>>>>> - >>>>>>> >>>>>>> Support data structures like lists, caped lists, hashes, sets >>>>>>> etc. (https://redis.io/topics/data-types-intro) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Fast read,write performances >>>>>>> 2. >>>>>>> >>>>>>> Provide persistence by writing to hard disk in configurable >>>>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>>>> 3. >>>>>>> >>>>>>> Can implement message queue structures. ( >>>>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>>>> e-queue-with-redis/ >>>>>>> >>>>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>>>> ) >>>>>>> 4. >>>>>>> >>>>>>> Use to implement message stores. (https://redis.io/topics/pubsub) >>>>>>> 5. >>>>>>> >>>>>>> In built basic transaction support (MULTI/EXEC commands : >>>>>>> https://redis.io/topics/transactions >>>>>>> <https://redis.io/topics/transactions>) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Loss of data (in case of power outage etc.) >>>>>>> 2. >>>>>>> >>>>>>> Depends on CPU performances when data amount is too high ( >>>>>>> https://redis.io/topics/persistence >>>>>>> <https://redis.io/topics/persistence>) >>>>>>> >>>>>>> RocksDB >>>>>>> >>>>>>> >>>>>>> - >>>>>>> >>>>>>> A simple key value storage (upgrade of LevelDB) >>>>>>> - >>>>>>> >>>>>>> In memory database >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Fast read,write performances (faster than LevelDB) >>>>>>> 2. >>>>>>> >>>>>>> Can implement queue structures (https://github.com/facebook/r >>>>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>>>> >>>>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>>>> ) >>>>>>> 3. >>>>>>> >>>>>>> Support concurrency >>>>>>> 4. >>>>>>> >>>>>>> Highly configurable (Pluggable Architecture) >>>>>>> 5. >>>>>>> >>>>>>> Support persistence (memtables and transactional logs manage in >>>>>>> memory. Memtable is flushed to a sstfile to provide persistence.) >>>>>>> >>>>>>> >>>>>>> 1. >>>>>>> >>>>>>> Loss of data (in case of power outage etc.) >>>>>>> 2. >>>>>>> >>>>>>> No in built transaction support (have to use TansactionDB : >>>>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>>>> >>>>>>> >>>>>>> According to this evaluation I suggest Redis as the most suitable >>>>>>> database for the message broker store. Even though it has an element of >>>>>>> risk in data loss, the persistence storing is configurable unlike in >>>>>>> other >>>>>>> key-value stores. And it provides fast read and write performances >>>>>>> compared >>>>>>> to other databases with basic transaction support. >>>>>>> >>>>>>> And this is basically my evaluation and the opinion. It would be >>>>>>> great to have a feedback on this, specially regarding the additional >>>>>>> criteria to consider and other suggested databases. >>>>>>> >>>>>>> Best Regards >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *Wishmitha Mendis* >>>>>>> >>>>>>> *Intern - Software Engineering* >>>>>>> *WSO2* >>>>>>> >>>>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Malaka. >>>>>> -- >>>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>>> Email : [email protected] >>>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>>> Web : http://wso2.com >>>>>> <http://wso2.com/signature> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Malaka. >>>>> -- >>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>> Email : [email protected] >>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>> Web : http://wso2.com >>>>> <http://wso2.com/signature> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sasikala Kottegoda >>>> *Senior Software Engineer* >>>> WSO2 Inc., http://wso2.com/ >>>> lean. enterprise. middleware >>>> Mobile: +94 774835928 <+94%2077%20483%205928> >>>> >>>> [image: https://wso2.com/signature] <https://wso2.com/signature> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Riyafa Abdul Hameed >>> Software Engineer, WSO2 Lanka (Pvt) Ltd <http://wso2.com/> >>> >>> Email: [email protected] <[email protected]> >>> Website: https://riyafa.wordpress.com/ <http://riyafa.wordpress.com/> >>> <http://facebook.com/riyafa.ahf> <http://lk.linkedin.com/in/riyafa> >>> <http://twitter.com/Riyafa1> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Asanka Abeyweera >> Senior Software Engineer >> WSO2 Inc. >> >> Phone: +94 712228648 <+94%2071%20222%208648> >> Blog: a5anka.github.io >> >> <https://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Manuri Amaya Perera* > > *Software Engineer* > > *WSO2 Inc.* > > *Blog: http://manuriamayaperera.blogspot.com > <http://manuriamayaperera.blogspot.com>* > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Wishmitha Mendis* *Intern - Software Engineering* *WSO2* *Mobile : +94 777577706*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
