[Dev] [MB][MQTT] By passing the message store for QoS 0 flows when running in stand alone mode
Hi All, Since QoS 0 in MQTT is a performance oriented delivery mode, an in-memory based message store was implemented to bypass the message database when running MB in stand alone mode, however when running in cluster in-memory store will not be applicable, since in order to exchange messages across the cluster, it requires a mechanism to communicate between brokers (Which will be introduced in future MB versions) In Stand-Alone mode the current message flow would be similar to the following, Publisher --- Message --- Deliver from In-Memory Store --- Deliver from Andes Kernal Store --- Message --- Subscription In the above flow, the subscriptions with QoS 0 would receive the messages through the in-memory store and the also currently the published message would be sent to the Andes kernal as well. Since for the same topic there could be subscriptions which are QoS 0. Making a call to Andes Kernal strore would not make much sense if there're no subscriptions for a particular topic which is QoS 0, also it will cause additional operation overhead. Therefore, we're in the process of implementing a solution which would be able to identify whether there are subscriptions QoS 0 for a given topic to identify the necessity to perform a call to the store. If we maintain the state of topics which has subscriptions QoS 0, for each message published it would require to iterate through all the subscriptions O(n), since there will be cases where the published topic will not 1:1 map with the subscribed topic, (i.e hierarchical topics with wildcards). In order to minimize the iteration cycles one option would be to partition subscriptions based on its attributes such as hierarchy, inclusive of wild card characters etc so that we could omit odds and iterate only through the possibilities. WDYT ? Please do advice on most optimized ways of filtering out matching subscriptions. Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB][MQTT] By passing the message store for QoS 0 flows when running in stand alone mode
Hi Pamod, Inverted bitmap technique[1] looks good for the purpose. Algorithm is O(n) where n is number of sections in the destination, not subscribers. Only concern is memory consumption and updating the maps when new subscribers come while messages are flowing through the system at higher rates (This will introduce some pauses to delivery). Since adding and removing subscriptions doesn't happen frequently we can live with that I think. Need to look at the memory consumption to make sure this is feasible. [1] http://zeromq.org/whitepapers:message-matching On Mon, Dec 15, 2014 at 5:52 PM, Pamod Sylvester pa...@wso2.com wrote: Hi All, Since QoS 0 in MQTT is a performance oriented delivery mode, an in-memory based message store was implemented to bypass the message database when running MB in stand alone mode, however when running in cluster in-memory store will not be applicable, since in order to exchange messages across the cluster, it requires a mechanism to communicate between brokers (Which will be introduced in future MB versions) In Stand-Alone mode the current message flow would be similar to the following, Publisher --- Message --- Deliver from In-Memory Store --- Deliver from Andes Kernal Store --- Message --- Subscription In the above flow, the subscriptions with QoS 0 would receive the messages through the in-memory store and the also currently the published message would be sent to the Andes kernal as well. Since for the same topic there could be subscriptions which are QoS 0. Making a call to Andes Kernal strore would not make much sense if there're no subscriptions for a particular topic which is QoS 0, also it will cause additional operation overhead. Therefore, we're in the process of implementing a solution which would be able to identify whether there are subscriptions QoS 0 for a given topic to identify the necessity to perform a call to the store. If we maintain the state of topics which has subscriptions QoS 0, for each message published it would require to iterate through all the subscriptions O(n), since there will be cases where the published topic will not 1:1 map with the subscribed topic, (i.e hierarchical topics with wildcards). In order to minimize the iteration cycles one option would be to partition subscriptions based on its attributes such as hierarchy, inclusive of wild card characters etc so that we could omit odds and iterate only through the possibilities. WDYT ? Please do advice on most optimized ways of filtering out matching subscriptions. Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB][MQTT] By passing the message store for QoS 0 flows when running in stand alone mode
Sure, let's discuss and see how we could incorporate this algorithm, This could also be added into the Andes Kernal as well. Thanks, Pamod On Tue, Dec 16, 2014 at 7:54 AM, Asitha Nanayakkara asi...@wso2.com wrote: Hi Pamod, Inverted bitmap technique[1] looks good for the purpose. Algorithm is O(n) where n is number of sections in the destination, not subscribers. Only concern is memory consumption and updating the maps when new subscribers come while messages are flowing through the system at higher rates (This will introduce some pauses to delivery). Since adding and removing subscriptions doesn't happen frequently we can live with that I think. Need to look at the memory consumption to make sure this is feasible. [1] http://zeromq.org/whitepapers:message-matching On Mon, Dec 15, 2014 at 5:52 PM, Pamod Sylvester pa...@wso2.com wrote: Hi All, Since QoS 0 in MQTT is a performance oriented delivery mode, an in-memory based message store was implemented to bypass the message database when running MB in stand alone mode, however when running in cluster in-memory store will not be applicable, since in order to exchange messages across the cluster, it requires a mechanism to communicate between brokers (Which will be introduced in future MB versions) In Stand-Alone mode the current message flow would be similar to the following, Publisher --- Message --- Deliver from In-Memory Store --- Deliver from Andes Kernal Store --- Message --- Subscription In the above flow, the subscriptions with QoS 0 would receive the messages through the in-memory store and the also currently the published message would be sent to the Andes kernal as well. Since for the same topic there could be subscriptions which are QoS 0. Making a call to Andes Kernal strore would not make much sense if there're no subscriptions for a particular topic which is QoS 0, also it will cause additional operation overhead. Therefore, we're in the process of implementing a solution which would be able to identify whether there are subscriptions QoS 0 for a given topic to identify the necessity to perform a call to the store. If we maintain the state of topics which has subscriptions QoS 0, for each message published it would require to iterate through all the subscriptions O(n), since there will be cases where the published topic will not 1:1 map with the subscribed topic, (i.e hierarchical topics with wildcards). In order to minimize the iteration cycles one option would be to partition subscriptions based on its attributes such as hierarchy, inclusive of wild card characters etc so that we could omit odds and iterate only through the possibilities. WDYT ? Please do advice on most optimized ways of filtering out matching subscriptions. Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [MB][MQTT] By passing the message store for QoS 0 flows when running in stand alone mode
This needs to be implemented for topic matching as well. On Tue, Dec 16, 2014 at 9:58 AM, Pamod Sylvester pa...@wso2.com wrote: Sure, let's discuss and see how we could incorporate this algorithm, This could also be added into the Andes Kernal as well. Thanks, Pamod On Tue, Dec 16, 2014 at 7:54 AM, Asitha Nanayakkara asi...@wso2.com wrote: Hi Pamod, Inverted bitmap technique[1] looks good for the purpose. Algorithm is O(n) where n is number of sections in the destination, not subscribers. Only concern is memory consumption and updating the maps when new subscribers come while messages are flowing through the system at higher rates (This will introduce some pauses to delivery). Since adding and removing subscriptions doesn't happen frequently we can live with that I think. Need to look at the memory consumption to make sure this is feasible. [1] http://zeromq.org/whitepapers:message-matching On Mon, Dec 15, 2014 at 5:52 PM, Pamod Sylvester pa...@wso2.com wrote: Hi All, Since QoS 0 in MQTT is a performance oriented delivery mode, an in-memory based message store was implemented to bypass the message database when running MB in stand alone mode, however when running in cluster in-memory store will not be applicable, since in order to exchange messages across the cluster, it requires a mechanism to communicate between brokers (Which will be introduced in future MB versions) In Stand-Alone mode the current message flow would be similar to the following, Publisher --- Message --- Deliver from In-Memory Store --- Deliver from Andes Kernal Store --- Message --- Subscription In the above flow, the subscriptions with QoS 0 would receive the messages through the in-memory store and the also currently the published message would be sent to the Andes kernal as well. Since for the same topic there could be subscriptions which are QoS 0. Making a call to Andes Kernal strore would not make much sense if there're no subscriptions for a particular topic which is QoS 0, also it will cause additional operation overhead. Therefore, we're in the process of implementing a solution which would be able to identify whether there are subscriptions QoS 0 for a given topic to identify the necessity to perform a call to the store. If we maintain the state of topics which has subscriptions QoS 0, for each message published it would require to iterate through all the subscriptions O(n), since there will be cases where the published topic will not 1:1 map with the subscribed topic, (i.e hierarchical topics with wildcards). In order to minimize the iteration cycles one option would be to partition subscriptions based on its attributes such as hierarchy, inclusive of wild card characters etc so that we could omit odds and iterate only through the possibilities. WDYT ? Please do advice on most optimized ways of filtering out matching subscriptions. Thanks, Pamod -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682 -- *Pamod Sylvester * *Senior Software Engineer * Integration Technologies Team, WSO2 Inc.; http://wso2.com email: pa...@wso2.com cell: +94 77 7779495 -- Ramith Jayasinghe Technical Lead WSO2 Inc., http://wso2.com lean.enterprise.middleware E: ram...@wso2.com P: +94 777542851 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
