Hi Youcef,

WSO2 Message broker has a thread pool with threads working to push messages
from DB to subscribers.
Thus if we create x number of connections to a broker those x number of
connections will be distributed across the threads in the pool.
We use LMax Disruptor [1] to batch incoming messages and write them to DB
as batches to gain performance.

There will be a upper bound for number of threads we can register to push
messages out. Also, I think one node can handle around 200-500 connections
(subscribers). We have not done proper stress testing on that side still.

Doing maths and assuming liner scaling, this will need 1M/500 = 2000 nodes
in cluster!  We cannot maintain such large cluster.
Thus suggestion would be to have a small set of clusters pointed to
different DBs.

My figures might be wrong. Maybe a MB node can handle more connections than
that. As a part of evaluation effort this is something needs to be done. If
these individual connections are idle most of the time, yes, then can have
more connections connected to a single node as threads would be idle and
available.

Another figure to consider is message distribution algorithm of MB needs to
make copies of  metadata+content of a single message one per node (if y
number of subscribers are there for that topic on that node, that y number
of subscribers will share that copy). Thus when number of nodes in a
cluster grows, this overhead also grows. This is also something to
consider.

Cannot say the requirement cannot be achieved, but we need to do a proper
evaluation as we have not tested MB on that number of connections.



[1]. https://lmax-exchange.github.io/disruptor/

Thanks

On Fri, Feb 2, 2018 at 6:41 PM, Youcef HILEM <youcef.hi...@laposte.fr>
wrote:

> Hi Imesh,
>
> Thank you very much.
>
> I must present the solutions to the architectural committee of Tuesday,
> 06/02/2018.
> It's good to have some information even this weekend on my work email.
>
>
>
> Thanks
> Youcef HILEM
>
>
>
> --
> Sent from: http://wso2-oxygen-tank.10903.n7.nabble.com/WSO2-
> Architecture-f62919.html
> _______________________________________________
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>



-- 
*Hasitha Abeykoon*
Associate Technical Lead; WSO2, Inc.; http://wso2.com
*cell:* *+94 719363063*
*blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Reply via email to