Hi Wishmitha, Since Redis is not an RDBMS it is interesting to know how the transactions are handled. When implementing distributed transactions feature we required row level locking instead of table level locking. We need to check if Redis also supports row level locking kind of feature. Further, in the article [1], it says Redis does not support rollbacks. Need to investigate more and see if this is a blocker for MB.
[1] https://redis.io/topics/transactions On Fri, Jul 14, 2017 at 3:21 PM, Maryam Ziyad <[email protected]> wrote: > Hi Wishmitha, All, > > It would be interesting to also look at how AOF persistence in Redis and > the append only journal used by Apache ActiveMQ Artemis [1] compare. > > Regarding AOF, fsync and durability, this > <http://oldblog.antirez.com/post/redis-persistence-demystified.html> blog > post pointed at by the official Redis site, even though a bit old, could be > useful. It mentions "Data committed to the disk using the fsync(2) system > call (or equivalent) that gives us, *virtually*, data safety against > complete system failure like a power outage." as a safety level [2]. > Also, out of the three options made available for fsync (appendfsync no, > appendfsync everysec and appendfsync always) they mention that "appendfsync > always" (with every query) is the best in terms of durability, but is > obviously slower compared to the other modes, and recommend "appendfsync > everysec" as a good balance between durability and speed [3]. > > Since these configurations seem straightforward and Redis has other useful > advantages as mentioned, doing a POC with Redis and testing seems like a > suitable approach. > > [1] https://activemq.apache.org/artemis/docs/2.1.0/persistence.html > [2] http://oldblog.antirez.com/post/redis-persistence-demystified.html - > "What we can't control" > [3] http://oldblog.antirez.com/post/redis-persistence-demystified.html - > "AOF durability" > > Best Regards, > Maryam > > On Fri, Jul 14, 2017 at 11:38 AM, Asanka Abeyweera <[email protected]> > wrote: > >> Hi Manuri, >> >> Rather than syncing periodically (1-second or less), they say they can >> sync at every query with AOF [1]. I don't' know the performance impact >> of that approach. Need to test and see. I am not proposing to use Redis. >> Just pointing the information. >> >> [1] https://redis.io/topics/persistence >> >> On Fri, Jul 14, 2017 at 10:27 AM, Manuri Amaya Perera <[email protected]> >> wrote: >> >>> Hi, >>> >>> From the given description, it seems persistence is supported by writing >>> to hard disk in configurable intervals. Unless we have a very low interval >>> (which might be against any expected performance improvement brought by the >>> database being in-memory) it cannot be assured that there won't be data >>> loss. Also even a low interval would not fully guarantee that. >>> I had a quick look at [1] and it says "You can configure Redis to have >>> it save the dataset every N seconds if there are at least M changes in the >>> dataset, or you can manually call the SAVE or BGSAVE commands.". So, if >>> server crashes we could lose 1 - M changes. We can only reduce it but >>> cannot avoid it completely. :) >>> So I'm not sure if we could use Redis. >>> >>> [1] https://redis.io/topics/persistence >>> >>> Thanks, >>> Manuri >>> >>> On Fri, Jul 14, 2017 at 9:39 AM, Asanka Abeyweera <[email protected]> >>> wrote: >>> >>>> Hi Wishmitha, >>>> >>>> If we use AOF wth fsync at every query, will there be a data loss with >>>> Redis? AOF might not have very good performance compared to RDB. But as >>>> Sasikal mentioned we cannot allow message losses in the broker. >>>> >>>> On Fri, Jul 14, 2017 at 8:42 AM, Riyafa Abdul Hameed <[email protected]> >>>> wrote: >>>> >>>>> Dear Wishmitha, >>>>> >>>>> For all the databases mentioned please provide references to the >>>>> relevant websites and where you got the information from. You have >>>>> mentioned for some which is great and you should do so for others as well. >>>>> Also when referencing don't have the weblink inline with the statement >>>>> because it distracts the viewer as the links might be long. You can number >>>>> the references and refer them below (You don't need to follow any >>>>> standards, simply giving the link would suffice). >>>>> >>>>> Cheers, >>>>> Riyafa >>>>> >>>>> On Fri, Jul 14, 2017 at 7:48 AM, Sasikala Kottegoda <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Wishmitha, >>>>>> >>>>>> We cannot tolerate data loss in the system. Guaranteed delivery comes >>>>>> first, performance comes next for the broker. >>>>>> >>>>>> Thank you, >>>>>> Sasikala >>>>>> >>>>>> On Fri, Jul 14, 2017 at 6:48 AM, Malaka Gangananda <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I have attached the pdf of above mentioned document. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> On Fri, Jul 14, 2017 at 6:38 AM, Malaka Gangananda <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Wishmitha, >>>>>>>> >>>>>>>> I have done some research work about existing message brokers >>>>>>>> previously, and here I have attach the link to the document which I >>>>>>>> have >>>>>>>> recorded the details about ActiveMQ Artemis high performance file >>>>>>>> journal >>>>>>>> and other file based message broker systems [1]. I think it will be >>>>>>>> useful >>>>>>>> to get an idea about how and why existing message brokers have >>>>>>>> implemented >>>>>>>> file based message stores. >>>>>>>> >>>>>>>> [1] https://docs.google.com/a/wso2.com/document/d/19CKLoAIZrue5O >>>>>>>> 047AnNOL6E24S4KcbNhWF81Mb3Id4E/edit?usp=sharing >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Thu, Jul 13, 2017 at 6:07 PM, Wishmitha Mendis < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I am working on a project to replace the current RDBMS based >>>>>>>>> database of the message broker store with a file based database >>>>>>>>> system. I >>>>>>>>> have been researching on potential file based databases which can be >>>>>>>>> used >>>>>>>>> in the project scenario. I have evaluated the pros and cons of the >>>>>>>>> each of >>>>>>>>> the potential databases mainly based on read and write performances, >>>>>>>>> transaction support, data structure (queue) implementation support and >>>>>>>>> overhead of replacement. >>>>>>>>> >>>>>>>>> A summary of database evaluation is represented in the following >>>>>>>>> table. >>>>>>>>> >>>>>>>>> >>>>>>>>> Database >>>>>>>>> >>>>>>>>> Pros >>>>>>>>> >>>>>>>>> Cons >>>>>>>>> >>>>>>>>> LevelDB >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> A simple key value storage >>>>>>>>> - >>>>>>>>> >>>>>>>>> Used in ActiveMQ (replaced with KahaDB) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Fast due to sorted keys >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Simple transaction support with batch( ) operation >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Sublevel feature (provides more organization and transactions >>>>>>>>> facilitation) >>>>>>>>> 4. >>>>>>>>> >>>>>>>>> LiveStream feature (able to query things which are yet being >>>>>>>>> receiving) >>>>>>>>> 5. >>>>>>>>> >>>>>>>>> Support triggers, groupby (mapreduce) and join (fanout) etc. >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Performance decrease in concurrent / multi threaded scenarios. >>>>>>>>> (improved in RocksDB.) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> No object relational mapping. >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Poor performance when memory is not enough. >>>>>>>>> >>>>>>>>> MongoDB >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> Document oriented >>>>>>>>> - >>>>>>>>> >>>>>>>>> Can define JSON schemas >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Direct object mapping (good replacement for RDBMS) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Can implement queue like data structures (store messages in >>>>>>>>> separate documents and delete them once delivered / already >>>>>>>>> implemented >>>>>>>>> data structures : https://www.npmjs.com/package/mongodb-queue) >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Capped collections (automatically remove old docs to make >>>>>>>>> space for new docs.) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> No transaction support for multiple documents. >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Low performance compared to other NoSQL databases. >>>>>>>>> >>>>>>>>> Caassandra >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> Column oriented (a key is denoted by row and an attribute is >>>>>>>>> denoted by a column which can dynamically change.) >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> Node - Cluster configuration (all nodes are identical) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> High availability (No single point of failure. In case of node >>>>>>>>> failure other identical nodes are available.) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> No data loss due to replication (identical nodes) >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Easy to model (column store is very similar to RDBMS tables.) >>>>>>>>> 4. >>>>>>>>> >>>>>>>>> Similar query language to SQL (CQL) >>>>>>>>> 5. >>>>>>>>> >>>>>>>>> High performance (No master node concept. Any node can provide >>>>>>>>> requested query. No performance bottleneck.) >>>>>>>>> 6. >>>>>>>>> >>>>>>>>> Transaction implementation >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Does not support queue structure (message queue implementation >>>>>>>>> is an anti-pattern : http://www.datastax.com/dev/bl >>>>>>>>> og/cassandra-anti-patterns-queues-and-queue-like-datasets >>>>>>>>> >>>>>>>>> <http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets> >>>>>>>>> ) >>>>>>>>> >>>>>>>>> Redis >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> A simple key value storage >>>>>>>>> - >>>>>>>>> >>>>>>>>> In memory database >>>>>>>>> - >>>>>>>>> >>>>>>>>> Support data structures like lists, caped lists, hashes, sets >>>>>>>>> etc. (https://redis.io/topics/data-types-intro) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Fast read,write performances >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Provide persistence by writing to hard disk in configurable >>>>>>>>> intervals. (snapshots : https://redis.io/topics/persistence) >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Can implement message queue structures. ( >>>>>>>>> http://fiznool.com/blog/2016/02/24/building-a-simple-messag >>>>>>>>> e-queue-with-redis/ >>>>>>>>> >>>>>>>>> <http://fiznool.com/blog/2016/02/24/building-a-simple-message-queue-with-redis/> >>>>>>>>> ) >>>>>>>>> 4. >>>>>>>>> >>>>>>>>> Use to implement message stores. (https://redis.io/topics/pubsu >>>>>>>>> b) >>>>>>>>> 5. >>>>>>>>> >>>>>>>>> In built basic transaction support (MULTI/EXEC commands : >>>>>>>>> https://redis.io/topics/transactions >>>>>>>>> <https://redis.io/topics/transactions>) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Loss of data (in case of power outage etc.) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Depends on CPU performances when data amount is too high ( >>>>>>>>> https://redis.io/topics/persistence >>>>>>>>> <https://redis.io/topics/persistence>) >>>>>>>>> >>>>>>>>> RocksDB >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> >>>>>>>>> A simple key value storage (upgrade of LevelDB) >>>>>>>>> - >>>>>>>>> >>>>>>>>> In memory database >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Fast read,write performances (faster than LevelDB) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> Can implement queue structures (https://github.com/facebook/r >>>>>>>>> ocksdb/wiki/Implement-Queue-Service-Using-RocksDB >>>>>>>>> >>>>>>>>> <https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB> >>>>>>>>> ) >>>>>>>>> 3. >>>>>>>>> >>>>>>>>> Support concurrency >>>>>>>>> 4. >>>>>>>>> >>>>>>>>> Highly configurable (Pluggable Architecture) >>>>>>>>> 5. >>>>>>>>> >>>>>>>>> Support persistence (memtables and transactional logs manage >>>>>>>>> in memory. Memtable is flushed to a sstfile to provide >>>>>>>>> persistence.) >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. >>>>>>>>> >>>>>>>>> Loss of data (in case of power outage etc.) >>>>>>>>> 2. >>>>>>>>> >>>>>>>>> No in built transaction support (have to use TansactionDB : >>>>>>>>> https://github.com/facebook/rocksdb/wiki/Transactions >>>>>>>>> <https://github.com/facebook/rocksdb/wiki/Transactions> ) >>>>>>>>> >>>>>>>>> >>>>>>>>> According to this evaluation I suggest Redis as the most suitable >>>>>>>>> database for the message broker store. Even though it has an element >>>>>>>>> of >>>>>>>>> risk in data loss, the persistence storing is configurable unlike in >>>>>>>>> other >>>>>>>>> key-value stores. And it provides fast read and write performances >>>>>>>>> compared >>>>>>>>> to other databases with basic transaction support. >>>>>>>>> >>>>>>>>> And this is basically my evaluation and the opinion. It would be >>>>>>>>> great to have a feedback on this, specially regarding the additional >>>>>>>>> criteria to consider and other suggested databases. >>>>>>>>> >>>>>>>>> Best Regards >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *Wishmitha Mendis* >>>>>>>>> >>>>>>>>> *Intern - Software Engineering* >>>>>>>>> *WSO2* >>>>>>>>> >>>>>>>>> *Mobile : +94 777577706 <+94%2077%20757%207706>* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Malaka. >>>>>>>> -- >>>>>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>>>>> Email : [email protected] >>>>>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>>>>> Web : http://wso2.com >>>>>>>> <http://wso2.com/signature> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malaka. >>>>>>> -- >>>>>>> Malaka Gangananda - Software Engineer | WSO2 >>>>>>> Email : [email protected] >>>>>>> Mobile : +94713564340 <+94%2071%20356%204340> >>>>>>> Web : http://wso2.com >>>>>>> <http://wso2.com/signature> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sasikala Kottegoda >>>>>> *Senior Software Engineer* >>>>>> WSO2 Inc., http://wso2.com/ >>>>>> lean. enterprise. middleware >>>>>> Mobile: +94 774835928 <+94%2077%20483%205928> >>>>>> >>>>>> [image: https://wso2.com/signature] <https://wso2.com/signature> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Riyafa Abdul Hameed >>>>> Software Engineer, WSO2 Lanka (Pvt) Ltd <http://wso2.com/> >>>>> >>>>> Email: [email protected] <[email protected]> >>>>> Website: https://riyafa.wordpress.com/ <http://riyafa.wordpress.com/> >>>>> <http://facebook.com/riyafa.ahf> <http://lk.linkedin.com/in/riyafa> >>>>> <http://twitter.com/Riyafa1> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Asanka Abeyweera >>>> Senior Software Engineer >>>> WSO2 Inc. >>>> >>>> Phone: +94 712228648 <+94%2071%20222%208648> >>>> Blog: a5anka.github.io >>>> >>>> <https://wso2.com/signature> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> *Manuri Amaya Perera* >>> >>> *Software Engineer* >>> >>> *WSO2 Inc.* >>> >>> *Blog: http://manuriamayaperera.blogspot.com >>> <http://manuriamayaperera.blogspot.com>* >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Asanka Abeyweera >> Senior Software Engineer >> WSO2 Inc. >> >> Phone: +94 712228648 <+94%2071%20222%208648> >> Blog: a5anka.github.io >> >> <https://wso2.com/signature> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Maryam Ziyad Mohamed* > Software Engineer | WSO2 > [image: http://wso2.com/signature] <http://wso2.com/signature> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Asanka Abeyweera Senior Software Engineer WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
