Hi Wishmitha, As per ActiveMQ [1] it says "The LevelDB store has been deprecated and is no longer supported or recommended for use. The recommended store is KahaDB" ... Are we aware of the reason LevelDB is not recommending in ActiveMQ? In that case are we on the correct path to use LevelDB ?
[1] http://activemq.apache.org/leveldb-store.html On Tue, Aug 15, 2017 at 10:47 AM, Wishmitha Mendis <[email protected]> wrote: > Hi Sumedha, > > The Java library for LevelDB (leveldbjni) creates the database as follows > as mentioned in the docs. [1] > > Options options = new Options(); > DB db = factory.open(new File("example"), options); > > This will create the database in a directory on a given path. And in the > library docs, it is mentioned that the library supports several platforms > if not specifically configured. Therefore using this library does not > require to ship LevelDB and it also won't take away platform agnostic > installation capability of MB. However the implementation is currently only > tested on Linux, I will test it on Windows and other platforms and let you > know. > > When considering the LevelDB architecture, it is already used as a broker > store in ActiveMQ. [2] [3] This proves that LevelDB has the architectural > capability to efficiently insert and delete messages in a broker. > > [1] https://github.com/fusesource/leveldbjni > [2] http://activemq.apache.org/leveldb-store.html > [3] https://github.com/apache/activemq/tree/master/activemq-leveldb-store > > Best Regards, > > On Tue, Aug 15, 2017 at 2:29 AM, Sumedha Rubasinghe <[email protected]> > wrote: > >> Hi Wishmitha, >> Would leveldb architecture be efficient for a message broker where >> removing delivered messages is very frequent? >> >> This requires WSO2 Message Broker to ship leveldb. leveldb ( >> https://github.com/google/leveldb) has native distributions for >> platforms. AFAIC this will take away platform agnostic installation >> capability of MB. >> >> >> >> On Tue, Aug 15, 2017 at 2:20 AM, Wishmitha Mendis <[email protected]> >> wrote: >> >>> Hi all, >>> >>> I am working on a project to replace the current RDBMS based database of >>> the message broker store with a file based database system. Currently the >>> implementation is carried out in LevelDB which is a key-value based data >>> store. The following is an explanation of suggested key schema for the data >>> store with related design decisions. >>> >>> *Overview :* >>> >>> LevelDB is a key value based database where a value can be stored under >>> a certain unique key. This key-value mapping is one directional which means >>> a value only can be retrieved by accessing corresponding key. One of the >>> main features in LevelDB is that it stores keys in a lexicographically >>> (alphabetically) sorted order. All the keys and values are stored in byte >>> array format in the database store which should be accordingly converted to >>> string format within the application. >>> >>> For this LevelDB store implementation leveldbjni-1.8[1] is used which >>> provides a Java based API for LevelDB by providing following main >>> functionalities. >>> >>> >>> 1. >>> >>> put(key,value) : stores given value under the provided key >>> 2. >>> >>> get(key) : returns corresponding value to the key >>> 3. >>> >>> delete(key) : deletes given key >>> 4. >>> >>> batch() : provides atomicity for the operations >>> 5. >>> >>> iterator() : traverse through the stored keys >>> >>> >>> When designing the key schema in Level DB the following factors are >>> mainly considered. >>> >>> >>> 1. >>> >>> Lexicographical order of the stored keys >>> 2. >>> >>> Traversing through the keys >>> 3. >>> >>> Data organization >>> >>> >>> *Key Schema :* >>> >>> The key schema implementation was carried out for following tables of >>> the current RDBMS database. >>> >>> [image: Screenshot from 2017-08-14 01-13-33.png] >>> >>> The key schema is mainly designed by analyzing implemented queries for >>> data retrieval and inserting in the RDBMS. The key schema for above three >>> tables is represented below table. >>> >>> >>> [image: Screenshot from 2017-08-15 02-11-24.png] >>> >>> *Key : Value* >>> >>> *Purpose* >>> >>> MESSAGE.$message_id.QUEUE_ID : queue_id >>> >>> Stores queue id of the message. >>> >>> MESSAGE.$message_id.DLC_QUEUE_ID : dlc_queue_id >>> >>> Stores dlc queue id of the message. >>> >>> MESSAGE.$message_id.MESSAGE_METADATA : message_metadata >>> >>> Stores metadata of the message. >>> >>> MESSAGE.$message_id.$content_offset.MESSAGE_CONTENT : message_content >>> >>> Stores message content for a given message offset of the message. >>> >>> QUEUE.$queue_id.QUEUE_NAME : queue_name >>> >>> Stores name of the queue under the id. >>> >>> QUEUE.$queue_name.QUEUE_ID : queue_id >>> >>> Stores id of the queue under the name. >>> >>> QUEUE.$queue_name.message_id. MESSAGE_METADATA : message_metadata >>> >>> Stores metadata of the messages which belongs to the queue. >>> >>> LAST_MESSAGE_ID >>> >>> Stores last message id. >>> >>> LAST_QUEUE_ID >>> >>> Stores last queue id. >>> >>> As it can be seen some data repetition is higher when using this schema. >>> That is mainly due to one directional key-value mapping of LevelDB. As an >>> example two keys (QUEUE.$queue_id.QUEUE_NAME , >>> QUEUE.$queue_name.QUEUE_ID) are required to build the bidirectional >>> relation (get queue name given queue id and get queue id given queue >>> name) between queue name and the queue id. As LevelDB has better >>> writing performances than RDBMS data repetition may not be an much of an >>> overhead in inserting data. Moreover batch operations can be used in >>> multiple insertions. >>> >>> The main purpose of using of prefixes like MESSAGE and QUEUE in keys is >>> to organize them properly. As LevelDB stores keys lexicographically these >>> prefixes will make sure that message related and queue related keys are >>> stored separately as displayed below. The following shows the keys of the >>> LevelDB store after publishing a JMS message to the broker. It can be >>> clearly seen that the keys are stored in lexicographical order. >>> >>> [image: Screenshot from 2017-08-14 19-57-13.png] >>> >>> Organize keys in such a manner also improves the efficiency of >>> traversing the keys using iterators when retrieving and deleting data. As >>> displayed in the diagram below, iterators traverse by starting from the >>> first stored key in the store. When iterator head reaches a key it can >>> either move to the next key or previous key. (similar to double linked >>> list) Hence storing related keys successively improves the efficiency >>> of traversing when retrieving and deleting data by reducing the seeking >>> time. >>> >>> >>> [image: Screenshot from 2017-08-15 02-11-40.png] >>> >>> >>> Basically these are the factors and decisions which have been taken in >>> implementing this key schema. And this schema should be extended to provide >>> functionalities like storing message expiration data etc. It would be great >>> to to have a feedback on the proposed schema specially regarding how to >>> reduce data repetition and improve efficiency furthermore. >>> >>> [1] https://github.com/fusesource/leveldbjni >>> >>> >>> Best Regards, >>> -- >>> >>> *Wishmitha Mendis* >>> >>> *Intern - Software Engineering* >>> *WSO2* >>> >>> *Mobile : +94 777577706 <077%20757%207706>* >>> >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > *Wishmitha Mendis* > > *Intern - Software Engineering* > *WSO2* > > *Mobile : +94 777577706 <+94%2077%20757%207706>* > > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sajith Kariyawasam *Associate Tech Lead* *WSO2 Inc.; http://wso2.com <http://wso2.com/>* *Committer and PMC member, Apache Stratos * *AMIE (SL)* *Mobile: 0772269575*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
