Miles, mesh-like architecture which uses CouchDB replication protocol to distribute/aggregate data is perfectly viable. As well as other members of the ML I also have several projects employing this approach. And like others I also can not disclose any details, even qty of nodes involved; you better be ready to see this attitude for any mesh-related project, because they often involve processing sensitive personal, industry or infrastructure data in vast amounts.
However, I have a lot of observations, main are: - whenever possible use CouchDB, not Pouch, because failure rate of Pouch instances on whatever platform is much higher overall - CouchDB is able to run for years without getting down, Pouch is not so sturdy, there exist a bunch of ways to knock down Pouch node with a single request - on mobile devices you better avoid constantly growing DBs, you may loose data and DB integrity eventually - a single doc rejected by Couch during upstream replication from Pouch can block entire process of replication, you can easily get the situation when sync just can’t restart – so carefully check what you store in Pouch, especially attachments size - get ready to spend a lot of time debugging Pouch—Couch routes having enormous latencies (like sync through satellites or 14400 GPRS connections) - mesh CouchDB networks tend to have very fast growing logs, reason is the number of repeated connections from different peers, so take care of it - do not distribute mesh topology across nodes using CouchDB replication, this approach looks inviting but brings a lot of risks. Best regards. ermouth вс, 24 мая 2020 г. в 18:54, Miles Fidelman <[email protected]>: > > On 5/24/20 11:42 AM, Jan Lehnardt wrote: > > > >> On 24. May 2020, at 17:32, Miles Fidelman <[email protected]> > wrote: > >> > >> Thanks Jan, and a follow-up, below: > >> > >> On 5/24/20 4:51 AM, Jan Lehnardt wrote: > >>>> On 23. May 2020, at 18:57, Miles Fidelman <[email protected]> > wrote: > >>>> > >>>> On 5/23/20 12:29 PM, Jan Lehnardt wrote: > >>>> > >>>>>> On 23. May 2020, at 16:28, Miles Fidelman < > [email protected]> wrote: > >>>>>> > >>>>>> > >>>>>> On 5/22/20 11:13 AM, Jan Lehnardt wrote: > >>>>>> > >>>>>>>> On 22. May 2020, at 15:06, Miles Fidelman < > [email protected]> wrote: > >>>>>>>> > >>>>>>>> Hi Jan, > >>>>>>>> > >>>>>>>> On 5/22/20 6:17 AM, Jan Lehnardt wrote: > >>>>>>>>> Hi Miles, > >>>>>>>>> > >>>>>>>>> I wanted to reply for a while, but struggled to find a good > angle. I think I finally figured out what I missed. I’m not sure I > understand your deployment scenario. > >>>>>>>>> > >>>>>>>>> When I think conference app, I think folks having that on their > mobile phones, or tablets. Given that, you’d be using PouchDB (for web > apps) or one of Cloudant mobile sync libraries (if it is a native app). > Either way, to get the data to the devices, it’d have to come from > somewhere, but then you say there is no central server. Where does the data > come from? > >>>>>>>> I was using that as an example - but I'm really thinking more the > case of "smart documents." Think of a plan (business, mission) or briefing > package - and the whole notion of "living documents. > >>>>>>> not sure what these are and how they compare to CouchDB documents. > >>>>>> "Living Document" as in writerspeak - a binder full of > documentation that is continually being kept up to date, the latest version > of a proposal, a book draft, a theatrical script that's being marked up > with commentary during rehearsal, etc. > >>>>>> > >>>>>> Or think of marrying a wiki to a DCVS. > >>>>>> > >>>>>>>>> It sounds like you imagine the devices talking to each other in > a replication mesh kind of way . While technically possible from a CouchDB > replication protocol point of view, this approach isn’t very practical. For > one, you won’t be able to guarantee that all devices can talk to each > other, especially when you don’t control the wifi. > >>>>>>>>> > >>>>>>>>> What’s more likely is that you’d have central CouchDB server > that is connected to the wifi network, either on site, or in a datacenter > somewhere that all devices connect to. > >>>>>>>>> > >>>>>>>>> Having that many devices talk to a central database to get all > the relatively static data of the conference (schedule, info, etc), doesn’t > sound like much of a problem. Group interaction is a little more > interesting. You could model this a 1-db-per-group, and it would work okay, > but there are some devil-in-the-details-details to work out with access > control and joining and leaving a group, etc. > >>>>>>>>> > >>>>>>>>> So overall: > >>>>>>>>> > >>>>>>>>> - what architecture do you envision? > >>>>>>>>> - I think this can be made to work. > >>>>>>>>> - CouchDB definitely can handle 5000 intermittent clients. > Depending on your use-case, you may need more or less computing resources > for this, but there aren’t any fundamental blockers in CouchDB’s design > that would prevent this from being a success. > >>>>>>>> I keep coming back to the model of replicated copies, stored in > something like a distributed version control system, linked by a > publish-subscribe network. (Not that much different than the way > large-scale modeling & sims are done - local "world models" linked by a > protocol like DIS or HLA.) > >>>>>>>> > >>>>>>>> I've been wondering if Couch/Pouch might make a good platform - > coupled with something like NNTP for epidemic style distribution of changes. > >>>>>>> If you squint a little, CouchDB fits your buzzword bill here, but > without more concrete descriptions of what you need (rather than less), we > can’t help much. > >>>>>>> > >>>>>>> And you didn’t address the network connectivity issue. If you plan > to have device-to-device communication, and you are not Apple or Google > making those devices and/or mobile operating systems, you won’t be having > much luck, especially on the web platform. > >>>>>> I'm thinking tactical environments & mesh networks, with > intermittent connectivity. Updates propagate slowly, but with either > automatic fork/merge, or with eventual consistency. > >>>>> Right, so far the theory, which the CouchDB replication protocol > supports. Now in practice, say you are using PouchDB, how do you make one > device’s browser talk to another device’s browser? > >>>> Well... I would see this as a case of mesh networking. An instance > of PouchDB, on a phone, might sync to one (or more) instances of CouchDB - > running on a laptop, or a server. The basic model would be that of NNTP. > Updates would ripple through a mesh of instances. My assumption is, that > any particular instance of Couch of Pouch needs to know one or more other > instances to connect to - and from there, stuff moves around. > >>> that is more concrete and we can talk about it now, thanks for > clarifying. > >>> > >>>> Which comes back to... how well does this scale? What's the largest > mesh network of Pouch/Couch instances that anybody has run in practice? > >>> since the setup is very non-standard, there isn’t a lot of existing > setups to look for. > >>> > >>> I know of a PouchDB-running-inside-node.js on IoT devices scenario > covering a large geographic area with mesh wifi, collecting data and > replicating sensory data through the mesh up to one point that has an > internet uplink, which then sends the whole bunch up to the cloud. At the > time I was involved, the target for this was 1000s of devices, but I know > little about the amount of documents flowing through the mesh. > >>> > >>> Again, the replication model fully support this, but given enough > data, the sensory data option might eventually be limited by data size / > storage space available on the devices, to hold, potentially, all data from > all devices. > >>> > >>> For your use-case, you have a lot less data coming in, so I wouldn’t > see much of an issue, overall. > >> > >> Do you, perhaps, have a reference for that implementation? > > I do not, my company consulted on this with a large technology company > on this. I’m not at liberty to disclose any details, or even names. > > > Ah well. Though I just spent some time nosing around "who's using > PouchDB" and there seem to be some folks doing large scale things. > Going to start digging in. (Searching on PouchDB +IoT got me there.) > > > Thanks! > > Miles > > > > > -- > > In theory, there is no difference between theory and practice. > > In practice, there is. .... Yogi Berra > > > > Theory is when you know everything but nothing works. > > Practice is when everything works but no one knows why. > > In our lab, theory and practice are combined: > > nothing works and no one knows why. ... unknown >
