Maybe someone else can shine some light here? I would be looking at a MQTT over websockets. (install mosquitto as a server) Using standard HTTPS/websocket upgrade will allow you to work on any network, trafic looks like normal https (even behind firewalls (normal ports open) and NAT) I find it works very reliably for what I need. (many devices, pub sub system, ACL for pieces of the hierarchy, low latency, simple to handle large payloads) I am not an expert, that is what I would do. Best regards, Edwin
On Tue, 3 Nov 2020 at 13:49, Yanone <p...@yanone.de> wrote: > > > > Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar < > ed...@oetelaar.com>: > > > > Define : *very* high number of subscribers > > It needs to be able to handle let's say <100K subscribers from the > beginning without changes. Later, that number could go up to many millions > when the big corporations jump onboard, which would be cool because then I > can generate funding for a tool that aims to level the power asymmetry > between them and the small and independent publishers. I want them to jump > onboard, so I need to be prepared. > > The design needs take this into account from the beginning. What this > means for me is that I could continue to work on the messaging > infrastructure without having to update the user-installed apps. > Sure, I *could* roll out updates, but I'd rather not break functionality > for those apps that are not getting updated for whatever reason. > > So I can postpone the actual scaling, but I feel like I need to make the > right decisions from the get-go. Any change to the messaging system on the > client side yields a functionality break. > > > > > > how about some RSS feed or NNTP ? > > I'm not sure how NNTP works exactly, but for sure constant polling is out > of question I believe (RSS is not a transport protocol). Because the > updates in my system will be very infrequent but need to trigger a somewhat > instant GUI app event when they do happen, frequent polling is out of > question. My whole system is designed rather distributed, so the > subscription update payload will be polled from the small publisher’s web > servers directly (after the discussed update notification) and I can't > burden them or my own central server with a high and growing polling load. > > Pub/Sub is perfect as long as the messaging system can handle high numbers > of subscribers. > > > > Or does it have to be real time in milli-seconds? > > No, up to a few seconds lag is fine. Like a messaging app – as long as you > can use it, it'll work. Messages can arrive a few seconds late and I can > still have a conversation. > > > > > Why ZeroMQ? > > Honestly, I'm really hoping that you guys can answer that question for me. > Really, that's why I'm here. > > I had already implemented Google’s commercial Pub/Sub, which works like a > charm for my app, but topics and subscribers per topic are both limited to > 10K each, so I can't even launch with that. AWS’s solution handles more, > but also limited. After there, you need to roll your own for specs alone, > plus it'll be cheaper in the long run. > > Thanks for your time. > Jan > > > > > > Best regards, > > Edwin #oetelx > > > > On Tue, 3 Nov 2020 at 12:51, Yanone <p...@yanone.de> wrote: > > Hello friends, > > > > I'm new to the world of messaging systems. I'm developing an open source > app in the world of graphic design that aims to change the way fonts are > installed on people’s computers and with that tool help small and > independent publishers to a better market position against the big evil > corporations in our industry. > > > > I need the GUI app to react to when either the user data or the font > subscription data has changed. The central server will inform the GUI app > of these changes and the app needs to react to them somewhat instantly. A > few seconds lag are no problem, neither are missing messages as I can > record the update timestamp for all datasets on the central server and have > the GUI app ask the central server for the timestamps on startup or > network-reconnect for all datasets that the GUI app instance is concerned > with. > > > > Back to ZeroMQ. > > So I need to design a pub/sub system that is horizontally scalable if > necessary. At first I'm sure I can do with a few tens of thousands of > subscribers, but later that number could grow dramatically. The thing > doesn't need to scale now, but eventually it may need to. So I'd rather > make the right decisions now. > > > > I read the guide (discarded non-pub/sub topics), and from what I > understand, I can do with the simple-most pub/sub pattern. I don't need to > store anything as the central server can keep the update timestamps as > described. Messages can go lost. Only as long as the client is running and > connected, I want updates to trigger. > > > > I'm expecting extremely little throughput (could be one update per topic > in many months), but I'm expecting high number of topics and *very* high > number of subscribers. Theoretically, two topics are sufficient (user > accounts + subscriptions) when applying filters. This will mean even higher > number of subscribers per topic. Otherwise, each user account and each > subscription gets their own topic. In that case, normal number of > subscribers per topic will be very low, except in rarer cases of free font > subscriptions, when a single subscription could still get a very high > number of subscribers. > > > > I haven't found *any* information anywhere on ZeroMQ limitations. So I > need to ask you for clarifications. > > > > Here are a precise set of questions: > > 1. What are the limitations of number of topics and number of > subscribers, either per topic or in total? > > 2. Which setup scales better? Only two topics with all subscribers in > them and filters, or subscribers spread out across a high number of topics? > In any case, the total number of subscribers stays the same. > > 3. In case of hard limitations on the above, how do I scale a cluster > horizontally? How are the subscribers and topics spread across containers > and brokered between them? > > 4. What is a good environment in the Google Cloud for that? I was eyeing > Kubernetes. I have no experience with it as I come from a Google App Engine > (Python) background, but I can learn it. The real question is: Will it > scale? > > > > Thank you so much for your input! 🙏 > > > > > > > > > > > > -- > > > > Yanone > > Dipl.-Designer, > > Master of Design in Type & Media > > https://yanone.de > > Twitter/Instagram: @yanone > > +49 176 24023268 (Germany) > > > > Type.World project: > > https://type.world > > > > Crowd funding on: > > https://www.patreon.com/typeWorld > > > > > > _______________________________________________ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > _______________________________________________ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > -- > > Yanone > Dipl.-Designer, > Master of Design in Type & Media > https://yanone.de > Twitter/Instagram: @yanone > +49 176 24023268 (Germany) > > Type.World project: > https://type.world > > Crowd funding on: > https://www.patreon.com/typeWorld > > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
