Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-11-22 Thread Wishmitha Mendis
Hi all, A File-Based DTX store was implemented using LevelDB to store distributed transaction related data. As XA transaction (two-phase commit) is used to handle the distributed transaction in MB, the related xid along with node id and branch id should be stored. Each transaction consists of messa

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-11-10 Thread Wishmitha Mendis
Hi all, The key schema for LevelDB store was updated due to following reasons. 1. Reduce data replication 2. Minimize the lookup time for iterators when retrieving data When updating the schema lexicographically ordered storage mechanism of LevelDB was taken into consideration. U

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-16 Thread Hasitha Hiranya
Hi Asanka and all, On Wed, Aug 16, 2017 at 12:08 PM, Asanka Abeyweera wrote: > Hi Wishmitha, > > On Tue, Aug 15, 2017 at 5:34 PM, Wishmitha Mendis > wrote: > >> Hi Asanka, >> >> 1. We can initially use a Network File System as you mentioned for HA. >> However data replication in LevelDB is used

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-16 Thread Pamod Sylvester
Hi Asanka, I also agree on the importance of the implementing a transaction mechanism. My concern is with key-value DB, we are implementing some capabilities (indexing, transactions, etc) which come out of the box with RDBMS. If we don't achieve a considerable amount of gain in performance, it wil

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-16 Thread Asanka Abeyweera
Hi Pamod, I also agree on the importance of the implementing a transaction mechanism. My concern is with key-value DB, we are implementing some capabilities (indexing, transactions, etc) which come out of the box with RDBMS. If we don't achieve a considerable amount of gain in performance, it will

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-16 Thread Pamod Sylvester
I suppose we just need to adhere to the scheme for indexing which is followed by on disk key-value dbs. Based on [1] MariaDB uses levelDB for it's internal storage engine. And also they've explained how indexing is handled in levelDB for it's use. Also i believe we don't need to stick with levelD

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-15 Thread Asanka Abeyweera
Hi Wishmitha, On Tue, Aug 15, 2017 at 5:34 PM, Wishmitha Mendis wrote: > Hi Asanka, > > 1. We can initially use a Network File System as you mentioned for HA. > However data replication in LevelDB is used in ActiveMQ replicated store. > [1] > Then It is better to run the performance test with a

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-15 Thread Wishmitha Mendis
Hi Asanka, 1. We can initially use a Network File System as you mentioned for HA. However data replication in LevelDB is used in ActiveMQ replicated store. [1] 2. Yes, the iterator should traverse all the related keys to get message data. This is why the key schema is designed in a such way that

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-15 Thread Wishmitha Mendis
Hi Sajith, Yes we are aware of the fact that ActiveMQ does not recommend LevelDB store currently. That is mainly because ActiveMQ is more focused on improving KahaDB which is the default database of their broker [1]. And they introduced LevelDB store mainly to increase the performances of the brok

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Asanka Abeyweera
Hi Wishmitha, 1. How are we going to support HA deployment with LevelDB? Are we going to use a network file system or replicate data? 2. If we wanted to get a set of message matching a given queue ID, do we have to traverse all messages to get that? In RDBMS this is easier to do wit

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Sajith Kariyawasam
Hi Wishmitha, As per ActiveMQ [1] it says "The LevelDB store has been deprecated and is no longer supported or recommended for use. The recommended store is KahaDB" ... Are we aware of the reason LevelDB is not recommending in ActiveMQ? In that case are we on the correct path to use LevelDB ? [1]

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Wishmitha Mendis
Hi Sumedha, The Java library for LevelDB (leveldbjni) creates the database as follows as mentioned in the docs. [1] Options options = new Options(); DB db = factory.open(new File("example"), options); This will create the database in a directory on a given path. And in the library docs, it is me

Re: [Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Sumedha Rubasinghe
Hi Wishmitha, Would leveldb architecture be efficient for a message broker where removing delivered messages is very frequent? This requires WSO2 Message Broker to ship leveldb. leveldb ( https://github.com/google/leveldb) has native distributions for platforms. AFAIC this will take away platform

[Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Wishmitha Mendis
Hi all, I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. Currently the implementation is carried out in LevelDB which is a key-value based data store. The following is an explanation of suggested key schema for the

[Architecture] Proposed Key Schema for LevelDB Store of Message Broker

2017-08-14 Thread Wishmitha Mendis
Hi all, I am working on a project to replace the current RDBMS based database of the message broker store with a file based database system. Currently the implementation is carried out in LevelDB which is a key-value based data store. The following is an explanation of suggested key schema for the