IMO, If we gonna keep huge messages as chunks in memory and insert into DB as bulk it will heavily affect on MB heap memory. My suggestion is we need to handle this case by case. For example, if it's small messages it will be efficient to keep in memory while huge messages it will be efficient to insert into DB early as possible and let others to use heap memory. For this we will have to make this functionality more configurable but again we will have to think about how gonna support fail over (probably have to change db schema).
*Dhanuka Ranasinghe* Senior Software Engineer WSO2 Inc. ; http://wso2.com lean . enterprise . middleware phone : +94 715381915 On Mon, Jul 21, 2014 at 9:00 AM, Asitha Nanayakkara <[email protected]> wrote: > We are planning to insert message chunks as batch insert queries. > > > On Sat, Jul 19, 2014 at 11:00 AM, Dhanuka Ranasinghe <[email protected]> > wrote: > >> Are we going to insert whole message or as chunks >> On 18 Jul 2014 18:06, "Asitha Nanayakkara" <[email protected]> wrote: >> >>> Hi, >>> >>> Following is the RDBMS design for WSO2 MB 3.0.0 >>> >>> Messages model >>> >>> Message metadata model >>> >>> >>> >>> Following are the concerns came across in the discussion >>> >>> *- Why we use reference counting for message meta data?* >>> >>> Reference counting is needed to delete topic messages from the database >>> reliably in a cluster deployment >>> >>> *- How to manage a large tables like Messages table?* >>> >>> for Messages table use database partitioning >>> >>> For Metadata queries there will be no SQL joins, hence even if the table >>> would grow large that won't be an issue. >>> >>> Inserts and delete operation can be done as batch operations. >>> >>> *- Following option to save metadata was rejected due to following >>> reasons* >>> >>> >>> >>> In the above design MB instance will create Node_Queue tables and >>> Topic_Node_Queue >>> tables when each node connects to a cluster. This design was rejected >>> due to following reasons. >>> It's DB admins tasks to create and delete tables. MB should not modify >>> schema when joining to the cluster. There will be instances where MB users >>> might not have privileges to create tables. >>> >>> *- Supporting several SQL implementations.* >>> >>> Since we are using simple SQL operations those will not become an issue. >>> >>> >>> >>> >>> Thanks >>> -- >>> *Asitha Nanayakkara* >>> Software Engineer >>> WSO2, Inc. http://wso2.com/ >>> Mob: + 94 77 85 30 682 >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Asitha Nanayakkara* > Software Engineer > WSO2, Inc. http://wso2.com/ > Mob: + 94 77 85 30 682 > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
