Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
Granted, the feasibility of an in place, functional and reliable distributed system may well prove the best argument for the centralized option in the end. Hi CMR, I'm sorry -- i'm still having trouble figuring out this statement. Did you mean that the *infeasibility* of a distributed system is an argument for a centralized option? Or did you mean that the feasibility of a centralized system is an argument for a centralized option? Or that the feasibility of a distributed system is an argument for a distributed system? apologies; feasibility was implied as feasibility, or lack there of Therefore the daunting prospect of actually designing and implementing a distributed media system that realizes our goals may, in the end, be the best argument against it; and thus for the centralized. I say may be because there may in fact exist opensource code that we could leverage effectively for this task; I'm just ignorant of what's available and of the respective pros/cons.. Cheers CMR --enter gratuitous quotation that implies my profundity here--
Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
On Fri, 1 Aug 2003, Ka-Ping Yee wrote: On Fri, 1 Aug 2003, zachary rosen wrote: A quick note - the decentralized system that is being proposed is NOT peer to peer. At the top, at the aggregator, it functions just the same as the centralized solution: One database, searchable and acessable by all - ie napster. The difference is how the metada gets to the central DB. Either it is aggregated from nodes, or required to be centrally submitted to bypass the technical hurdles of aggregation. I believe we are all agreed up to this point. Woo! :) I am under the impression that feasability or technical-hard-ness of building the metadata collection functionality to be decentralized rather than requiring it to be centrally submitted does not nearly outweigh the problems with admining, maintaining, and hosting the central solution. We have to maintain the central host anyway, because we've already agreed that we need a main search site. The question is whether we ought to introduce *additional* administrative tasks for people running other DeanSpace sites in the media network. I completely disagree. I don't see how having node admins help manage their local repositories increases the amount (*additional*) administrative work. We want to have all nodes have media hosting capabilities (I believe). This means there will have to be admin work done to set up a media hosting module no matter what. The only real difference between centralized and decentralized in terms of admin work required then becomes the mundane maintenance tasks: pruning and organization. If the nodes are empowered to maintain their own local repository then doing this work is offloaded from the potentially very large bottlneck of having it all maintained / pruned in a central site by one set of admins (DMT) to the many capable node admins. Am I missing something? When i get home i will be rifling through some books for quotes to support my claim that Reed's Law and End-to-End principals support the decentraralized design over the centralized design. Since we seem to have different perspectives on what these terms mean, i'd rather talk directly about the reasoning, as in This design is better for the following reasons... rather than This design is better because there's a principle that says so. Of course ;p Now, i'm not against fault-tolerance. Yes, of course it would be good to have a resilient system where some of the hosts can fail and things keep working. We can certainly talk about ways to do that after the data has been centrally submitted (e.g. more traditional kinds of redundancy, such as mirror sites for the search engine). With the decentralized solution fault-tolerance is vastly easier to manage: it is built into the system - every node that does media stuff directly offloads hosting tasks from the central server. Furthermore, when problems arise the decentralized solution handles them far more gracefully. Even if the central aggregator goes down the network is still mostly functional - the only thing that breaks is central search. If the central site goes down then the entire system is basically sunk. It seems to me that the technical hurdles of designing reduncancy into the central solution to deal with these very real problems would be harder and require more engineering effort than creating the network decentralized in the first place. Furthermore the decentralzied solution can handle these problems enumerably better. -Zack
Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
On Fri, Aug 01, 2003 at 02:32:53PM -0500, zachary rosen wrote: On Fri, 1 Aug 2003, zachary rosen wrote: The only real difference between centralized and decentralized in terms of admin work required then becomes the mundane maintenance tasks: pruning and organization. If the nodes are empowered to maintain their own local repository then doing this work is offloaded from the potentially very large bottlneck of having it all maintained / pruned in a central site by one set of admins (DMT) to the many capable node admins. Am I missing something? Yeah. Anybody can prune the database. For maintenance operations that might be a little dangerous, like vetoing media, we can hand out moderator accounts on the central site to volunteers we trust. In fact, it would be easier to find volunteers to just moderate than volunteers to run an entire DeanSpace site. This is exactly the reason I am so opposed to this solution. It is a basic question: who do you trust more to vett / prune media on the system that comes from nodes? DMT - or the nodes themselves? It might be productive, at that juncture, to make explicit the assumptions you're carrying about what *sort* of vetting might be being done, by whom, and for what reasons. I suspect we have almost a *classic* case of the centralized/decentralized debate going here, and the two sides of this one aren't ever *going* to agree in my experience, so let's just yell Hitler! and Godwin!, and define our assumptions. :-) Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows -- Simon Slavin, on a.f.c
Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
A quick note - the decentralized system that is being proposed is NOT peer to peer. At the top, at the aggregator, it functions just the same as the centralized solution: One database, searchable and acessable by all - ie napster. Think I got it; so the aggregator functions like the news feeds(?) except it's remotely querying databases, periodically (or on the fly -pushed?-), The difference is how the metada gets to the central DB. Either it is aggregated from nodes, or required to be centrally submitted to bypass the technical hurdles of aggregation. The killer app aspect, then, is the automation of data directory centralization as opposed to relying on participants to visit and update the central site manually(?) I am under the impression that feasability or technical-hard-ness of building the metadata collection functionality to be decentralized rather than requiring it to be centrally submitted does not nearly outweigh the problems with admining, maintaining, and hosting the central solution. Given the above clarification, it sounds more do-able if the node admins can be educated.. When i get home i will be rifling through some books for quotes to support my claim that Reed's Law and End-to-End principals support the decentraralized design over the centralized design. Interestingly, reed gave a presentation where he illustrated his law (I've got a semantic bone to pick with that term as applied to theories * strict definitional sense*, but I digress..) using the example of online auctions: But the theory is less important than the practice, at least if you're trying to profit from the Internet, so I'll make some predictions based on the likely effects of the Group-Forming Law in 2002: The obvious conclusion is that whoever forms the biggest, most robust communities will win. But the Group-Forming idea can be used to look well beyond the obvious and discriminate among strategies that are all billed as building communities. For instance, Internet auction pioneer Onsale, which buys closeout products and auctions them on its Web site, will see its value rise only in proportion to the number of users. On-line classifieds, which connect buyers to sellers on a peer-to-peer basis, should see a stronger, Metcalfe effect. Ebay, which began as one person's attempt to establish a market for Pez candy dispensers, should get an even more powerful Group-Forming effect because it helps members act in groups as they auction off and bid for products on-line. (Other economics work in favor of Ebay, too. Because the Group-Forming effect will give it enormous volumes of business, it can charge a lower commission on sales. The low fees will attract more users and produce a virtuous circle. Also, because it's Ebay's customers who do the selling, Ebay doesn't face any inventory or product-development issues.) Notice he touts EBay as an example; a centralized system. But it doesn't necessarily follow that a thriving Group-Forming online communty can't be fostered via a distibuted network. Jon Udell apprently thinks just the opposite in evaluting the future of Radio Userland: Both approaches are valid, but there is a middle ground -- more coherent than email, less isolated than Groove -- that needs to be occupied. Radio doesn't yet know how to occupy that middle ground. But it has the tools people need to do the experiment: a distributed scripting engine and object database, Web-services protocols. When Radio's currently-centralized community engine itself becomes distributable (as is planned), I expect to see an explosion of group-forming activity. The spaces thus constituted will express different sets of values, but they'll federate in the way that Reed's Law predicts. see: http://www.xml.com/pub/a/ws/2002/03/01/udell.html
Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
On Fri, 1 Aug 2003, Ka-Ping Yee wrote: On Fri, 1 Aug 2003, zachary rosen wrote: This is exactly the reason I am so opposed to this solution. It is a basic question: who do you trust more to vett / prune media on the system that comes from nodes? DMT - or the nodes themselves? We are all on the same team, Zack. Site admins can volunteer to be moderators if they want; DMT people can volunteer to be moderators if they want; my Mom could volunteer to be a moderator, except that she has no idea what this is about (Oh, Howard who? That's nice.). It doesn't matter whether submission is central or distributed. But a separate issue: it sounds as though now you're suggesting that the media items sit on the nodes waiting for approval before they're forwarded up to the search database. Yet the earlier response to questions about update consistency was that things would travel automatically, two hops up. So which is it: friction or no friction? What bothers me about trying to discuss the distributed submission design is that we don't *have* a distributed submission design yet. If the hypothetical design keeps morphing each time there's a new question, that makes it difficult to talk about. -- ?!ng At the very least the system we come up with should support the following vetting functionality (in my opinion). 1] Nodes should be able to vett the media in their repositories 2] The central aggregator should be able to vett the media accessible in the central repository. With the central solution [1] becomes hard if not impossible to accomplish. Either a) every node is responsible for sending an admin to sign up and be given permission by the DMT to vet media for their node on the central site. or b) we build in functionality that allows nodes to veto media in their local repository that is culled from the central DB... which only solves half the problem: they still have no way to counter veto a vett from the DMT. Either solution will take many man hours of time to accomplish, and this would not even be an issue to consider with the decentralized solution (both the aggregators and the nodes automatically can vet to their hearts content). Whether it is frictionless or there is friction does not effect the engineering effort required to create this system if it is decentralized. If it is centralized then this DOES become a problem - because ultimately we would want to give the nodes the power to choose (frictionless or friction) and this would become extremely tricky if not impossible if we did make it centralized. -Zack
Re: Fw: [hackers] Re: Edge-to-Edge Principal / Reed's Law; revised2
On Fri, 1 Aug 2003, Ka-Ping Yee wrote: On Fri, 1 Aug 2003, zachary rosen wrote: 1] Nodes should be able to vett the media in their repositories 2] The central aggregator should be able to vett the media accessible in the central repository. With the central solution [1] becomes hard if not impossible to accomplish. Either a) every node is responsible for sending an admin to sign up and be given permission by the DMT to vet media for their node on the central site. What's so hard? In either case, a moderator opens her browser, logs into a site, browses the new media items, and rates them up or down. I'm going to stop talking about this for now and get back to coding. What is hard is human nature. If we create a system in which communities do not have a say in how their own sites are run - we bring on a whole slew of political / social problems. Nodes should be able to control their own media repositories period - not DMT. -Zack
