Le lun. 20 juil. 2020 à 03:57, Lansana Camara <[email protected]> a écrit : > > Can you provide a diagram of the architecture?
A DHT backend could be used to spread the on-disk load over many peers, but only as _cold storage_ not as a primary source of truth. > I'm having a hard time visualizing high-performance servers being wired up by > a database. I am not sure I understand that. Every system requires some kind of database. What could be done is to have a Single Source of Truth where everything is persisted forever and secondary sources of truth where expert systems (high-performance services) store the data in their favorite schema. This is usually the case of Event Sourcing systems. The whole system is eventually consistent. That means that a given expert system will have to wait sometime before they become aware of new knowledge. The advantage is that you keep around all the data in the log, the event source, and aggregate the useful knowledge for a given system in a fine-tuned highly performant database for specific uses. This is not my favorite approach but it is workable. > The DHT is just meant to be a storage mechanism, right? If so, why/how is it > wiring up servers? Seems like something is missing in this picture. For more info on DHT look at https://github.com/amirouche/qadom/blob/master/qadom/peer.py In particular, the routing mechanism is described in: https://github.com/amirouche/qadom/blob/b1bf5762ee62a958f5c9e9ea991d6f7c792fbece/qadom/peer.py#L305-L321 the nearest procedure is defined at: https://github.com/amirouche/qadom/blob/b1bf5762ee62a958f5c9e9ea991d6f7c792fbece/qadom/peer.py#L56-L62 That procedure is a hack to avoid the use of more involving data structure that is implemented in kademlia: https://github.com/bmuller/kademlia But it works in most situations when there are less than 1M peers in the network. Here is the procedure that fetches a key from the network: https://github.com/amirouche/qadom/blob/b1bf5762ee62a958f5c9e9ea991d6f7c792fbece/qadom/peer.py#L611-L658 Hope this helps! > On Sun, Jul 19, 2020 at 1:35 PM Linas Vepstas <[email protected]> wrote: >> >> >> As a "distributed atomspace" is a commonly recurring request, I thought I >> should put together yet-another distributed atomspace variant. This one is >> very super-simple: easy to use, and has a tiny implementation (500 lines of >> code). >> >> Easy-to-use: Take a look at the two examples, here: >> https://github.com/opencog/atomspace-cog/tree/master/examples >> >> These examples use exactly the same API as the 3-4 other distributed >> atomspace backends. Some comments about those. >> >> * The Postgres SQL backend. It's part of the core AtomSpace code, provided >> by default. Large, complex, bullet-proof, production-ready. Kind-of slow: >> there's a lot of overhead. >> >> * The IPFS backend. Sounds great, right? I tried a very very naive >> implementation, and it's surprisingly terrible. Turns out IPFS is actually >> "centralized" not "decentralized": to get anything done, one must build an >> index, that index must fit into just one file. Whoops. So clearly, my naive >> design is the wrong way to go. The code "works" (passes unit tests) but is >> disappointing. https://github.com/opencog/atomspace-ipfs >> >> * The OpenDHT backend. Taking the lessons learned above, I ported the same >> naive implementation to OpenDHT. So, DHT stands for "Distributed Hash >> Table", in this case Kademlia, the same one as in bittorrent and ethereum >> and gnunet and many others. Much better, but revealed a different flaw in my >> naive thinking. Two problems, harder to explain. Problem #1: an Atom, >> sitting in OpenDHT, is just taking up RAM, thus competing for RAM with any >> local AtomSpace. Problem #2: the hash used by DHT's completely randomizes >> Atoms. So even if they are close to one-another, e.g. (List (Concept "a") >> (Concept "b")) -- these three atoms - the two concepts and the list, will >> end up on different servers on opposite sides of the planet. The DHT hashing >> algo has no clue about the locality-of-reference that we want for the >> atomspace. Again, this code "works" (passes unit tests) but is >> disappointing. https://github.com/opencog/atomspace-dht >> >> * So what's the right design? Well, it seems that the best bet would be to >> use OpenDHT to store AtomSpace indexes, but do the actual serving of atoms >> by "seeders". And so this is why I wrote this super-simple cogserver-based >> distributed atomspace. The hope is to use it as a "seeder" >> https://github.com/opencog/atomspace-cog/ >> >> Future plans: I'm hoping that someone interested can build a >> high-performance server/seeder, based on the prototype here. (really -- 500 >> LOC is very simple, very easy to understand, and thus easy to improve upon.) >> This does NOT require any special skills: if you have basic coding skills, >> maybe some experience with network i/o, or are willing to explore, it >> should be possible to build a high-performance variation thereof. So all >> those people saying "I'm just an ordinary coder, how can I help?" well -- >> here's your chance. >> >> A more difficult, more conceptual task would be how to wire up a bunch of >> these servers using the OpenDHT/Kademlia infrastructure. I think this is >> possible, but it's more cerebral, and requires thinking-work. >> >> --linas >> >> -- >> Verbogeny is one of the pleasurettes of a creatific thinkerizer. >> --Peter da Silva >> >> -- >> You received this message because you are subscribed to the Google Groups >> "opencog" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/opencog/CAHrUA37wx%2B3SAtBS45eS6vf_XrC8rgCCdNW2HziA7xkYasCw_A%40mail.gmail.com. > > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/CAPPXERqiYtxJ1ucnkqMQLWmwo3OCy894J2OgYruE_dM7MQAjnw%40mail.gmail.com. -- Amirouche ~ https://hyper.dev -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAL7_Mo8B8p%2BZnTt0cgBgdqY%2B2gEa9_RydD2tyMwrxeU%3D6eCj6Q%40mail.gmail.com.
