On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[email protected]> wrote:
> Hi Manuri, > > Rather than syncing periodically (1-second or less), they say they can > sync at every query with AOF [1]. I don't' know the performance impact of > that approach. Need to test and see. I am not proposing to use Redis. Just > pointing the information. > Right.. Thanks for the information. I missed what had been said about AOF before replying. > > [1] https://redis.io/topics/persistence > > On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[email protected]> > wrote: > >> Hi, >> >> From the given description, it seems persistence is supported by writing >> to hard disk in configurable intervals. Unless we have a very low interval >> (which might be against any expected performance improvement brought by the >> database being in-memory) it cannot be assured that there won't be data >> loss. Also even a low interval would not fully guarantee that. >> I had a quick look at [1] and it says "You can configure Redis to have it >> save the dataset every N seconds if there are at least M changes in the >> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if >> server crashes we could lose 1 - M changes. We can only reduce it but >> cannot avoid it completely. :) >> So I'm not sure if we could use Redis. >> >> [1] https://redis.io/topics/persistence >> >> Thanks, >> Manuri >> >> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[email protected]> >> wrote: >> >>> Hi Wishmitha, >>> >>> If we use AOF wth fsync at every query, will there be a data loss with >>> Redis? AOF might not have very good performance compared to RDB. But as >>> Sasikal mentioned we cannot allow message losses in the broker. >>> >>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[email protected]> >>> wrote: >>> >>>> Dear Wishmitha, >>>> >>>> For all the databases mentioned please provide references to the >>>> relevant websites and where you got the information from. You have >>>> mentioned for some which is great and you should do so for others as well. >>>> Also when referencing don't have the weblink inline with the statement >>>> because it distracts the viewer as the links might be long. You can number >>>> the references and refer them below (You don't need to follow any >>>> standards, simply giving the link would suffice). >>>> >>>> Cheers, >>>> Riyafa >>>> >>>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[email protected]> >>>> wrote: >>>> >>>>> Hi Wishmitha, >>>>> >>>>> We cannot tolerate data loss in the system. Guaranteed delivery comes >>>>> first, performance comes next for the broker. >>>>> >>>>> Thank you, >>>>> Sasikala >>>>> >>>>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I have attached the pdf of above mentioned document. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Wishmitha, >>>>>>> >>>>>>> I have done some research work about existing message brokers >>>>>>> previously, and here I have attach the link to the document which I have >>>>>>> recorded the details about ActiveMQ Artemis high performance file >>>>>>> journal >>>>>>> and other file based message broker systems [1]. I think it will be >>>>>>> useful >>>>>>> to get an idea about how and why existing message brokers have >>>>>>> implemented >>>>>>> file based message stores. >>>>>>> >>>>>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >>>>>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I am working on a project to replace the current RDBMS based >>>>>>>> database of the message broker store with a file based database >>>>>>>> system. I >>>>>>>> have been researching on potential file based databases which can be >>>>>>>> used >>>>>>>> in the project scenario. I have evaluated the pros and cons of the >>>>>>>> each of >>>>>>>> the potential databases mainly based on read and write performances, >>>>>>>> transaction support, data structure (queue) implementation support and >>>>>>>> overhead of replacement. >>>>>>>> >>>>>>>> A summary of database evaluation is represented in the following >>>>>>>> table. >>>>>>>> >>>>>>>> >>>>>>>> Database >>>>>>>> >>>>>>>> Pros >>>>>>>> >>>>>>>> Cons >>>>>>>> >>>>>>>> LevelDB >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> A simple key value storage >>>>>>>> - >>>>>>>> >>>>>>>> Used in ActiveMQ (replaced with KahaDB) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Fast due to sorted keys >>>>>>>> 2. >>>>>>>> >>>>>>>> Simple transaction support with batch( ) operation >>>>>>>> 3. >>>>>>>> >>>>>>>> Sublevel feature (provides more organization and transactions >>>>>>>> facilitation) >>>>>>>> 4. >>>>>>>> >>>>>>>> LiveStream feature (able to query things which are yet being >>>>>>>> receiving) >>>>>>>> 5. >>>>>>>> >>>>>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>>>>> (improved in RocksDB.) >>>>>>>> 2. >>>>>>>> >>>>>>>> No object relational mapping. >>>>>>>> 3. >>>>>>>> >>>>>>>> Poor performance when memory is not enough. >>>>>>>> >>>>>>>> MongoDB >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Document oriented >>>>>>>> - >>>>>>>> >>>>>>>> Can define JSON schemas >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Direct object mapping (good replacement for RDBMS) >>>>>>>> 2. >>>>>>>> >>>>>>>> Can implement queue like data structures (store messages in >>>>>>>> separate documents and delete them once delivered / already >>>>>>>> implemented >>>>>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>>>>> 3. >>>>>>>> >>>>>>>> Capped collections (automatically remove old docs to make space >>>>>>>> for new docs.) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> No transaction support for multiple documents. >>>>>>>> 2. >>>>>>>> >>>>>>>> Low performance compared to other NoSQL databases. >>>>>>>> >>>>>>>> Caassandra >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Column oriented (a key is denoted by row and an attribute is >>>>>>>> denoted by a column which can dynamically change.) >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Node - Cluster configuration (all nodes are identical) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> High availability (No single point of failure. In case of node >>>>>>>> failure other identical nodes are available.) >>>>>>>> 2. >>>>>>>> >>>>>>>> No data loss due to replication (identical nodes) >>>>>>>> 3. >>>>>>>> >>>>>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>>>>> 4. >>>>>>>> >>>>>>>> Similar query language to SQL (CQL) >>>>>>>> 5. >>>>>>>> >>>>>>>> High performance (No master node concept. Any node can provide >>>>>>>> requested query. No performance bottleneck.) >>>>>>>> 6. >>>>>>>> >>>>>>>> Transaction implementation >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Does not support queue structure (message queue implementation >>>>>>>> is an anti-pattern : http://www.datastax.com/dev/bl >>>>>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>>>>> >>>>>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>>>>> ) >>>>>>>> >>>>>>>> Redis >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> A simple key value storage >>>>>>>> - >>>>>>>> >>>>>>>> In memory database >>>>>>>> - >>>>>>>> >>>>>>>> Support data structures like lists, caped lists, hashes, sets >>>>>>>> etc. (https://redis.io/topics/data-types-intro) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Fast read,write performances >>>>>>>> 2. >>>>>>>> >>>>>>>> Provide persistence by writing to hard disk in configurable >>>>>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>>>>> 3. >>>>>>>> >>>>>>>> Can implement message queue structures. ( >>>>>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>>>>> e-queue-with-redis/ >>>>>>>> >>>>>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>>>>> ) >>>>>>>> 4. >>>>>>>> >>>>>>>> Use to implement message stores. (https://redis.io/topics/pubsub >>>>>>>> ) >>>>>>>> 5. >>>>>>>> >>>>>>>> In built basic transaction support (MULTI/EXEC commands : >>>>>>>> https://redis.io/topics/transactions >>>>>>>> <https://redis.io/topics/transactions>) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Loss of data (in case of power outage etc.) >>>>>>>> 2. >>>>>>>> >>>>>>>> Depends on CPU performances when data amount is too high ( >>>>>>>> https://redis.io/topics/persistence >>>>>>>> <https://redis.io/topics/persistence>) >>>>>>>> >>>>>>>> RocksDB >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> A simple key value storage (upgrade of LevelDB) >>>>>>>> - >>>>>>>> >>>>>>>> In memory database >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Fast read,write performances (faster than LevelDB) >>>>>>>> 2. >>>>>>>> >>>>>>>> Can implement queue structures (https://github.com/facebook/r >>>>>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>>>>> >>>>>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>>>>> ) >>>>>>>> 3. >>>>>>>> >>>>>>>> Support concurrency >>>>>>>> 4. >>>>>>>> >>>>>>>> Highly configurable (Pluggable Architecture) >>>>>>>> 5. >>>>>>>> >>>>>>>> Support persistence (memtables and transactional logs manage in >>>>>>>> memory. Memtable is flushed to a sstfile to provide persistence.) >>>>>>>> >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>>>>> Loss of data (in case of power outage etc.) >>>>>>>> 2. >>>>>>>> >>>>>>>> No in built transaction support (have to use TansactionDB : >>>>>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>>>>> >>>>>>>> >>>>>>>> According to this evaluation I suggest Redis as the most suitable >>>>>>>> database for the message broker store. Even though it has an element of >>>>>>>> risk in data loss, the persistence storing is configurable unlike in >>>>>>>> other >>>>>>>> key-value stores. And it provides fast read and write performances >>>>>>>> compared >>>>>>>> to other databases with basic transaction support. >>>>>>>> >>>>>>>> And this is basically my evaluation and the opinion. It would be >>>>>>>> great to have a feedback on this, specially regarding the additional >>>>>>>> criteria to consider and other suggested databases. >>>>>>>> >>>>>>>> Best Regards >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *Wishmitha Mendis* >>>>>>>> >>>>>>>> *Intern - Software Engineering* >>>>>>>> *WSO2* >>>>>>>> >>>>>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malaka. >>>>>>> -- >>>>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>>>> Email : [email protected] >>>>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>>>> Web : http://wso2.com >>>>>>> <http://wso2.com/signature> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Malaka. >>>>>> -- >>>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>>> Email : [email protected] >>>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>>> Web : http://wso2.com >>>>>> <http://wso2.com/signature> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sasikala Kottegoda >>>>> *Senior Software Engineer* >>>>> WSO2 Inc., http://wso2.com/ >>>>> lean. enterprise. middleware >>>>> Mobile: +94 774835928 <+94%2077%20483%205928> >>>>> >>>>> [image: https://wso2.com/signature] <https://wso2.com/signature> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Riyafa Abdul Hameed >>>> Software Engineer, WSO2 Lanka (Pvt) Ltd <http://wso2.com/> >>>> >>>> Email: [email protected] <[email protected]> >>>> Website: https://riyafa.wordpress.com/ <http://riyafa.wordpress.com/> >>>> <http://facebook.com/riyafa.ahf> <http://lk.linkedin.com/in/riyafa> >>>> <http://twitter.com/Riyafa1> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Asanka Abeyweera >>> Senior Software Engineer >>> WSO2 Inc. >>> >>> Phone: +94 712228648 <+94%2071%20222%208648> >>> Blog: a5anka.github.io >>> >>> <https://wso2.com/signature> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> *Manuri Amaya Perera* >> >> *Software Engineer* >> >> *WSO2 Inc.* >> >> *Blog: http://manuriamayaperera.blogspot.com >> <http://manuriamayaperera.blogspot.com>* >> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Asanka Abeyweera > Senior Software Engineer > WSO2 Inc. > > Phone: +94 712228648 <+94%2071%20222%208648> > Blog: a5anka.github.io > > <https://wso2.com/signature> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Manuri Amaya Perera* *Software Engineer* *WSO2 Inc.* *Blog: http://manuriamayaperera.blogspot.com <http://manuriamayaperera.blogspot.com>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
