Hi, According to mail arch@ "Removing Global Queue from MB", maybe we need to update this design. Better we discuss upfront if so.
It will bring changes to messageStore interface. Thanks On Thu, Jul 24, 2014 at 3:04 PM, Asitha Nanayakkara <[email protected]> wrote: > With the addition of message expiration feature RDBMS design for Metadata > in MB needs to be changed. > Updated design is as follows. > > > > > * Design considerations* > > - Only a subset of messages comes with message expiration and a > separate thread handles deletion of expired messages. > - Periodically expired message deletion thread queries for a chunk of > expired messages. The subset of messages to go through would be less when a > separated Expiration table is used. However when deleting a message from > Expiration table should trigger a delete of all the messages with the > respective message_id from Metadata table. > > In addition, to update reference count of messages an SQL trigger will be > created. Is there any performance hit using triggers? > > > > > On Thu, Jul 24, 2014 at 12:06 PM, Asitha Nanayakkara <[email protected]> > wrote: > >> With current design in memory message store will be used only in single >> node mode. >> >> >> On Wed, Jul 23, 2014 at 11:36 AM, Dhanuka Ranasinghe <[email protected]> >> wrote: >> >>> Also, normally publisher mention whether to persist or not messages in >>> message itself (delivery mode). So based on that MB will process messages >>> in memory and/or persist to a persistence store. So if it's process in >>> memory How do we communicate through the MB cluster? >>> >>> >>> http://activemq.apache.org/what-is-the-difference-between-persistent-and-non-persistent-delivery.html >>> >>> >>> *Dhanuka Ranasinghe* >>> >>> Senior Software Engineer >>> WSO2 Inc. ; http://wso2.com >>> lean . enterprise . middleware >>> >>> phone : +94 715381915 >>> >>> >>> On Mon, Jul 21, 2014 at 12:36 PM, Dhanuka Ranasinghe <[email protected]> >>> wrote: >>> >>>> IMO, If we gonna keep huge messages as chunks in memory and insert into >>>> DB as bulk it will heavily affect on MB heap memory. My suggestion is we >>>> need to handle this case by case. For example, if it's small messages it >>>> will be efficient to keep in memory while huge messages it will be >>>> efficient to insert into DB early as possible and let others to use heap >>>> memory. For this we will have to make this functionality more configurable >>>> but again we will have to think about how gonna support fail over (probably >>>> have to change db schema). >>>> >>>> *Dhanuka Ranasinghe* >>>> >>>> Senior Software Engineer >>>> WSO2 Inc. ; http://wso2.com >>>> lean . enterprise . middleware >>>> >>>> phone : +94 715381915 >>>> >>>> >>>> On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara <[email protected]> >>>> wrote: >>>> >>>>> We are planning to insert message chunks as batch insert queries. >>>>> >>>>> >>>>> On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe <[email protected] >>>>> > wrote: >>>>> >>>>>> Are we going to insert whole message or as chunks >>>>>> On 18 Jul 2014 18:06, "Asitha Nanayakkara" <[email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Following is the RDBMS design for WSO2 MB 3.0.0 >>>>>>> >>>>>>> Messages model >>>>>>> >>>>>>> Message metadata model >>>>>>> >>>>>>> >>>>>>> >>>>>>> Following are the concerns came across in the discussion >>>>>>> >>>>>>> *- Why we use reference counting for message meta data?* >>>>>>> >>>>>>> Reference counting is needed to delete topic messages from the >>>>>>> database reliably in a cluster deployment >>>>>>> >>>>>>> *- How to manage a large tables like Messages table?* >>>>>>> >>>>>>> for Messages table use database partitioning >>>>>>> >>>>>>> For Metadata queries there will be no SQL joins, hence even if the >>>>>>> table would grow large that won't be an issue. >>>>>>> >>>>>>> Inserts and delete operation can be done as batch operations. >>>>>>> >>>>>>> *- Following option to save metadata was rejected due to following >>>>>>> reasons* >>>>>>> >>>>>>> >>>>>>> >>>>>>> In the above design MB instance will create Node_Queue tables and >>>>>>> Topic_Node_Queue >>>>>>> tables when each node connects to a cluster. This design was >>>>>>> rejected due to following reasons. >>>>>>> It's DB admins tasks to create and delete tables. MB should not >>>>>>> modify schema when joining to the cluster. There will be instances >>>>>>> where MB >>>>>>> users might not have privileges to create tables. >>>>>>> >>>>>>> *- Supporting several SQL implementations.* >>>>>>> >>>>>>> Since we are using simple SQL operations those will not become an >>>>>>> issue. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> -- >>>>>>> *Asitha Nanayakkara* >>>>>>> Software Engineer >>>>>>> WSO2, Inc. http://wso2.com/ >>>>>>> Mob: + 94 77 85 30 682 >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Asitha Nanayakkara* >>>>> Software Engineer >>>>> WSO2, Inc. http://wso2.com/ >>>>> Mob: + 94 77 85 30 682 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Asitha Nanayakkara* >> Software Engineer >> WSO2, Inc. http://wso2.com/ >> Mob: + 94 77 85 30 682 >> >> > > > -- > *Asitha Nanayakkara* > Software Engineer > WSO2, Inc. http://wso2.com/ > Mob: + 94 77 85 30 682 > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
