Re: [agi] Recap/Summary/Thesis Statement
Agents don't decide the goals of the group. I don't know about you but I tend to be an alpha male. I *DO* decide the goals of the groups that I'm in. :-) If I weren't such a studly alpha male, I would still decide goals of the group and then convince the group to go along with me/my goals.;-) You simply aren't envisioning smart enough agents. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: Agents don't decide the goals of the group. I don't know about you but I tend to be an alpha male. I *DO* decide the goals of the groups that I'm in. :-) If I weren't such a studly alpha male, I would still decide goals of the group and then convince the group to go along with me/my goals.;-) You simply aren't envisioning smart enough agents. I'm sure the agents can be very smart, but the group as a whole will be smarter than any of its members, and a set of competing groups with evolution as a judge will be smarter than any one group. For example, democracies tend to be more successful than dictatorships (which depend on the intelligence of its most successful member), but we wouldn't know that without war. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
I don't know about you but I tend to be an alpha male. I *DO* decide the goals of the groups that I'm in. :-) If I weren't such a studly alpha male, I would still decide goals of the group and then convince the group to go along with me/my goals.;-) You simply aren't envisioning smart enough agents. I'm sure the agents can be very smart, but the group as a whole will be smarter than any of its members, and a set of competing groups with evolution as a judge will be smarter than any one group. Yes, the group as a whole will be smarter than any of it's members because it will cherry-pick the best ideas of it's members. Survival is a good idea. An agent will come up with it. The group will adopt it. For example, democracies tend to be more successful than dictatorships (which depend on the intelligence of its most successful member), but we wouldn't know that without war. I entirely disagree with your hypothesis. I do believe that war sped up the process but we would have learned the lesson without them. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: I'm sure the agents can be very smart, but the group as a whole will be smarter than any of its members, and a set of competing groups with evolution as a judge will be smarter than any one group. Yes, the group as a whole will be smarter than any of it's members because it will cherry-pick the best ideas of it's members. This, and averaging decisions together are well known techniques in machine learning. Survival is a good idea. An agent will come up with it. The group will adopt it. It *seems* blatantly obvious that survival is a good idea. But good only has meaning with respect to a goal. Survival is good because those agents who didn't think so died, not because they came up with the idea. Agents don't choose their goals. Agents may very well come up with the idea that self sacrifice for the survival of the group is good for the group, but if that is not their goal to begin with, they aren't going to practice it. They actually have to be born with that goal and be selected by killing off competing groups. You might say that since we are building the agents, we can give them any goals we want. Suppose we give them goals of self sacrifice for the group, while humans remain selfish. Then humans will exploit them and they die. However if we give them policing powers (i.e. they can kill unFriendlies) then we die. You could give humans special status, but that goal is unstable in an evolutionary environment. For example, democracies tend to be more successful than dictatorships (which depend on the intelligence of its most successful member), but we wouldn't know that without war. I entirely disagree with your hypothesis. I do believe that war sped up the process but we would have learned the lesson without them. That is called hindsight bias. http://www.singinst.org/upload/cognitive-biases.pdf -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
It *seems* blatantly obvious that survival is a good idea. But good only has meaning with respect to a goal. Survival is good because those agents who didn't think so died, not because they came up with the idea. Agents don't choose their goals. Huh? I'm an agent and I choose my goals. If I live long enough to be in this super-society of yours, I think that I'm still going to have the goal of survival for both me and my super-society. You might say that since we are building the agents, we can give them any goals we want. We're not building the agents. I *am* the agent. I entirely disagree with your hypothesis. I do believe that war sped up the process but we would have learned the lesson without them. That is called hindsight bias. http://www.singinst.org/upload/cognitive-biases.pdf The tendency to believe that what actually happened had a higher probability than the actual real probability of it happening is called hindsight bias. Humans do indeed tend to have hindsight bias and I am no different; however, it is very rare that a human will claim 100% when the actual probability is 0%. You are claiming 0%. I am claiming, given sufficient time, 100%. It is unlikely that hindsight bias accounts for this major a difference. You have provided absolutely no proof for your hypothesis. You have simply called my hypothesis a name (hindsight bias) that is unlikely to actually apply. I have provided absolutely no proof for my hypothesis -- so I will admit that we're even. Care to try again? --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: It *seems* blatantly obvious that survival is a good idea. But good only has meaning with respect to a goal. Survival is good because those agents who didn't think so died, not because they came up with the idea. Agents don't choose their goals. Huh? I'm an agent and I choose my goals. If I live long enough to be in this super-society of yours, I think that I'm still going to have the goal of survival for both me and my super-society. You did not choose to fear death. You did not choose to not want to turn off your fear of death. Since everyone dies, don't you think it would be in your best interest to change your goals and avoid some anxiety? But evolution doesn't work that way. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
You did not choose to fear death. You did not choose to not want to turn off your fear of death. (I wondered if you would see that argument :-) OK. But I do have a goal of survival and I *WILL* pass it on to my super-society. Since everyone dies, don't you think it would be in your best interest to change your goals and avoid some anxiety? No. Anxiety is good for my goals. That's why evolution gave it to me (It's an Omohundro drive :-). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
You don't have a goal of self preservation. You have goals like eating, breathing, avoiding pain, etc. that increase the odds of passing on your genes. Wrong. I most certainly *DO* have a goal of self-preservation. Even if it is quick and utterly painless, I do *NOT* want to die. Why do you write such blatantly incorrect things? --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: You don't have a goal of self preservation. You have goals like eating, breathing, avoiding pain, etc. that increase the odds of passing on your genes. Wrong. I most certainly *DO* have a goal of self-preservation. Even if it is quick and utterly painless, I do *NOT* want to die. That is a learned goal, like acquiring money. It is not a top level goal. When you were a child and did not know you would die someday, did you fear death or did you fear the hundreds of things that might kill you? Learned goals can be reprogrammed, unlike top level goals. Would you fear your quick and painless destruction in a teleportation booth if an exact copy of you with all your memories was constructed at the other end? Why do you write such blatantly incorrect things? Because there is a problem with your design and you still don't see it. Where do top level goals come from? A group of friendly agents working together is the same thing as one vastly more intelligent agent. It will have some set of goals, but what? If it has no competition, then where is the selective pressure to maintain goals that promote self preservation? -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
If there are competing groups of agents, then evolution will favor goals that promote survival of the group (for example, self sacrifice). If there is only one group, then this evolution does not occur. Why not? The agents are competing with the environment (not each other) and will therefore evolve and get smarter. Any sufficiently smart agent will realize that it is in it's society's best interest (for the purposes of fulfilling it's goals -- whatever they are) to have a survival goal. Therefore, any sufficiently smart agent will GIVE it's society a survival goal. You need competition between groups to evolve strategies to defeat cancer. Huh? So you claim that a single cooperative group of super-geniuses can't evolve strategies to defeat cancer unless there is a competing group? That appears to be yet another blatantly incorrect Mahoneyism (a provably incorrect *opinion* stated as if it is an established fact). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: If there are competing groups of agents, then evolution will favor goals that promote survival of the group (for example, self sacrifice). If there is only one group, then this evolution does not occur. Why not? The agents are competing with the environment (not each other) and will therefore evolve and get smarter. Any sufficiently smart agent will realize that it is in it's society's best interest (for the purposes of fulfilling it's goals -- whatever they are) to have a survival goal. Therefore, any sufficiently smart agent will GIVE it's society a survival goal. Agents don't decide the goals of the group. The cells in your body did not decide to program goals like hunger into your brain. Likewise, individuals in a tribe or country do not decide that their culture ought to promote nationalistic pride or taboos on sexual activity for purposes other than reproduction. Instead, the members made random choices and competition between groups decides which goals survive. If you just have one group, then it will just have random goals. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: This is not the risk that concerns me. The real risk is that a single, fully cooperating system has no evolutionary drive for self improvement. So we provide an artificial evolutionary drive for the components of society via a simple economy . . . . as has been suggested numerous times by Baum and others. I looked up Baum's research, in particular http://www.whatisthought.com/hayek32000.pdf So what provides the artificial evolutionary drive in your system? I can accept that cooperation between agents and killing defectors (your definition of Friendly) leads to greater fitness, but where does the group goal of self preservation come from if there are no wars between groups? Suppose you were the only person on earth and you decided to stop eating? Your cells are cooperating perfectly with each other, for example, killing off cancerous mutations. But deciding to stop eating is a group decision. As long as there is competition between humans, this behavior is weeded out. Are you implying that Friendly behavior requires wars between groups, or is there something else that stops the whole group from making a bad collective decision? -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
So what provides the artificial evolutionary drive in your system? I can accept that cooperation between agents and killing defectors (your definition of Friendly) leads to greater fitness, but where does the group goal of self preservation come from if there are no wars between groups? From all of my other goals. If I die, none of them get done. This is why self-preservation is an Omohundro drive. Suppose you were the only person on earth and you decided to stop eating? I would die and all my goals would not be fulfilled. Therefore, as long as I have sufficiently important goals, I am *not* going to decide to stop eating (unless I am insufficiently intelligent/informed to know the fatal results of not eating). Your cells are cooperating perfectly with each other, for example, killing off cancerous mutations. But deciding to stop eating is a group decision. As long as there is competition between humans, this behavior is weeded out. Are you implying that Friendly behavior requires wars between groups, or is there something else that stops the whole group from making a bad collective decision? Absolutely not. Friendly behavior simply requires that your recognize sub-optimal behavior as getting in the way of *your* goals -- and specifies what some of that sub-optimal behavior is. Mark Friendliness: The Ice-9 of Ethics and Ultimate in Self-Interest --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: So what provides the artificial evolutionary drive in your system? I can accept that cooperation between agents and killing defectors (your definition of Friendly) leads to greater fitness, but where does the group goal of self preservation come from if there are no wars between groups? From all of my other goals. If I die, none of them get done. This is why self-preservation is an Omohundro drive. Which is not quite correct. Humans and other animals don't have Omohundro drives as explicit goals. They have goals that contribute to these drives. You don't have a goal of self preservation. You have goals like eating, breathing, avoiding pain, etc. that increase the odds of passing on your genes. This seems like a minor point but it is not. You cannot automatically assume that an agent in a cooperative environment wants to live. Indeed, if there is competition among groups, the most successful groups will have agents with goals that promote the survival of the group, not themselves. So I ask again. If there is just *one* group of cooperating agents, then where does the drive for group self preservation come from? I suppose that a group would have inherited goals that contribute to survival from an earlier time when groups competed. But if there is no competition, then those goals don't evolve. It is like humans given unlimited food, machines to do all our work, and drugs to make us happy. We evolved in a time when those things didn't exist, so we become fat, lazy addicts. How do you prevent your group of friendly agents from wireheading itself? -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
On 07/03/2008, Matt Mahoney [EMAIL PROTECTED] wrote: --- Mark Waser [EMAIL PROTECTED] wrote: Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels Proof: something will happen resulting in zero or more intelligent agents. Those agents will be Friendly to each other and themselves, because the action of killing agents without replacement is an irreversible dynamic, and therefore cannot be part of an attractor. Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) An attractor is a set of states that are repeated given enough time. If agents are killed and not replaced, you can't return to the current state. False. There are certainly attractors that disappear, first seen by Ruelle, Takens, 1971 its called a blue sky catastrophe http://www.scholarpedia.org/article/Blue-sky_catastrophe --linas --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
An attractor is a set of states that are repeated given enough time. If agents are killed and not replaced, you can't return to the current state. False. There are certainly attractors that disappear, first seen by Ruelle, Takens, 1971 its called a blue sky catastrophe http://www.scholarpedia.org/article/Blue-sky_catastrophe Relatedly, you should look at Mikhail Zak's work on terminal attractors, which occurred in the context of neural nets as I recall These are attractors which a system zooms into for a while, then after a period of staying in them, it zooms out of them They occur when the differential equation generating the dynamical system displaying the attractor involves functions with points of nondifferentiability. Of course, you may be specifically NOT looking for this kind of attractor, in your Friendly AI theory ;-) -- Ben --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
On 11/03/2008, Ben Goertzel [EMAIL PROTECTED] wrote: An attractor is a set of states that are repeated given enough time. If agents are killed and not replaced, you can't return to the current state. False. There are certainly attractors that disappear, first seen by Ruelle, Takens, 1971 its called a blue sky catastrophe http://www.scholarpedia.org/article/Blue-sky_catastrophe Relatedly, you should look at Mikhail Zak's work on terminal attractors, which occurred in the context of neural nets as I recall These are attractors which a system zooms into for a while, then after a period of staying in them, it zooms out of them That is how one would describe the classic and well-studied homoclinic orbit -- zoom in for a while then zoom out. They occur when the differential equation generating the dynamical system displaying the attractor involves functions with points of nondifferentiability. homoclinic orbits don't need non-differentibility; just saddle points, where the stable and unstable mainfolds join at right angles. Even with differentiable systems there's a dozen types of attractors, bifurcations (attractors which split in two) and the like; only one is the attracting fixed point that seems to be what the original poster was thinking of when he posted. Of course, you may be specifically NOT looking for this kind of attractor, in your Friendly AI theory ;-) Remember that attractors are the language of low-dimensional chaos, where there's only 3 or 4 variables. In neural nets, you have hundreds or more (gasp!) neurons, and so you are well out of the area where low-dimensional chaos theory applies, and in a whole new regime (turbulence, in physics), which is pretty much not understood at all in any branch of science. Of course, we just paint artistic impressions on this list, so this is hardly science... --linas --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
As of now, we are aware of no non-human friendlies, so the set of excluded beings will in all likelihood be the empty set. Eliezer's current vision of Friendliness puts AGIs (who are non-human friendlies) in the role of excluded beings. That is why I keep hammering this point. To answer your question, I don't see the people are evil and will screw it all up scenario as being even remotely likely, for reasons of self-interest among others. And I think it very likely that if it turns out that including non-human friendlies is the right thing to do, that the system will do as designed and renormalize accordingly. People are *currently* screwing it all up in the sense that our society is *seriously* sub-optimal and far, FAR less than it could be. Will we screw it up to the point of self-destruction? That's too early to tell. The Cuban Missile Crisis was an awfully near miss. Grey Goo would be *really* bad (though I think that it is a bit further off than most people on this list). It's scary to even consider what I *know* that I could do if I were a whack-job terrorist but with my knowledge. The only reason why I am as optimistic as I am currently is because I truly do believe that Friendliness is an attractor that we are solidly on the approach path to and I hope that I can speed the process by pointing that fact out. As for the other option, my question was not about the dangers relating to *who is or is not protected*, but rather *whose volition is taken into account* in calculating the CEV, since your approach considers only the volition of friendly humanity (and non-human friendlies but not non-friendly humanity), while Eliezer's includes all of humanity. Actually, I *will* be showing that basically Friendly behavior *IS* extended to everyone except in so far as non-Friendlies insist upon being non-Friendly. I just didn't see a way to successfully introduce that idea early *AND* forestall Vladimir's obvious so why don't I just kill them all argument. I need to figure out a better way to express that earlier. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Linas Vepstas [EMAIL PROTECTED] wrote: On 07/03/2008, Matt Mahoney [EMAIL PROTECTED] wrote: --- Mark Waser [EMAIL PROTECTED] wrote: Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels Proof: something will happen resulting in zero or more intelligent agents. Those agents will be Friendly to each other and themselves, because the action of killing agents without replacement is an irreversible dynamic, and therefore cannot be part of an attractor. Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) An attractor is a set of states that are repeated given enough time. If agents are killed and not replaced, you can't return to the current state. False. There are certainly attractors that disappear, first seen by Ruelle, Takens, 1971 its called a blue sky catastrophe http://www.scholarpedia.org/article/Blue-sky_catastrophe Also the simple point attractor x = 0 in the dynamical system dx/dt = -x with solution exp(-t) never repeats. But dynamical systems with real-valued states are just approximations of a discrete universe, and a discrete system must repeat. But I have a different objection to Mark's proposal: the only attractor in an evolutionary system is a dead planet. Evolution is not a stable system. It is on the boundary between stability and chaos. Evolution is punctuated by mass extinctions as well as smaller disasters, plagues, and population explosions. Right now I believe we are in the midst of a mass extinction larger than the Permian extinction. There are two reasons why I think we are still alive today: the anthropic principle, and a range of environments wide enough that no species can inhabit all of them (until now). Omohundro's goals are stable in an evolutionary system (as long as that system persists) because they improve fitness. In Mark's proposal, Friendliness is a subgoal to fitness because (if I understand correctly) agents that cooperate with each other are fitter as a group than agents that fight among themselves. So an outcome where the Earth is turned into a Dyson sphere of gray goo would be Friendly in the sense that the biggest army of nanobots kills off all their unFriendly competition (including all DNA based life) and they cooperate with each other. This is not the risk that concerns me. The real risk is that a single, fully cooperating system has no evolutionary drive for self improvement. Having one world government with perfect harmony among its population is a bad idea because there is no recourse from it making a bad collective decision. In particular, there is no evolutionary pressure to maintain a goal of self preservation. You need competition between countries, but unfortunately this means endless war. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
This is not the risk that concerns me. The real risk is that a single, fully cooperating system has no evolutionary drive for self improvement. So we provide an artificial evolutionary drive for the components of society via a simple economy . . . . as has been suggested numerous times by Baum and others. Really Matt, all your problems seem toi be due to a serious lack of imagination rather than pointing out actual contradictions or flaws. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
It *might* get stuck in bad territory, but can you make an argument why there is a *significant* chance of that happening? Not off the top of my head. I'm just playing it better safe than sorry since, as far as I can tell, there *may* be a significant chance of it happening. Also, I'm not concerned about it getting *stuck* in bad territory, I am more concerned about just transiting bad territory and destroying humanity on the way through. One thing that I think most of will agree on is that if things did work as Eliezer intended, things certainly could go very wrong if it turns out that the vast majority of people -- when smarter, more the people they wish they could be, as if they grew up more together ... -- are extremely unfriendly in approximately the same way (so that their extrapolated volition is coherent and may be acted upon). Our meanderings through state space would then head into very undesirable territory. (This is the people turn out to be evil and screw it all up scenario.) Your approach suffers from a similar weakness though, since it would suffer under the seeming friendly people turn out to be evil and screw it all up before there are non-human intelligent friendlies to save us scenario. But my approach has the advantage that it proves that Friendliness is in those evil people's self-interest so *maybe* we can convert them before they do us in. I'm not claiming that my approach is perfect or fool-proof. I'm just claiming that it's better than anything else thus far proposed. Which, if either, of 'including all of humanity' rather than just 'friendly humanity', or 'excluding non-human friendlies (initially)' do you see as the greater risk? I see 'excluding non-human friendlies (initially)' as a tremendously greater risk. I think that the proportionality aspect of Friendliness will keep the non-Friendly portion of humanity safe as we move towards Friendliness. Actually, let me rephrase your question and turn it around -- Which, if either, of 'not protecting all of humanity from Friendlies rather than just friendly humanity' or 'being actively unfriendly' do you see as a greater risk? Or is there some other aspect of Eliezer's approach that especially concerns you and motivates your alternative approach? The lack of self-reinforcing stability under errors and/or outside forces is also especially concerning and was my initially motivation for my vision. Thanks for continuing to answer my barrage of questions. No. Thank you for the continued intelligent feedback. I'm disappointed by all the people who aren't interested in participating until they can get a link to the final paper without any effort. This is still very much a work in progress with respect to the best way to present it and the only way I can improve it is with decent feedback -- which is therefore *much* appreciated. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
I've just carefully reread Eliezer's CEV http://www.singinst.org/upload/CEV.html, and I believe your basic idea is realizable in Eliezer's envisioned system. The CEV of humanity is only the initial dynamic, and is *intended* to be replaced with something better. I completely agree with these statements. It is Eliezer's current initial trajectory that I strongly disagree with (believe to be seriously sub-optimal) since it is in the OPPOSITE direction of where I see Friendliness. Actually, on second thought, I disagree with your statement that The CEV is only the initial dynamic. I believe that it is the final dynamic as well. A better phrasing that makes my point is that Eliezer's view of the CEV of humanity is only the initial dynamic and is intended to be replaced with something better. My claim is that my view is something better/closer to the true CEV of humanity. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
On 03/09/2008 10:20 AM,, Mark Waser wrote: My claim is that my view is something better/closer to the true CEV of humanity. Why do you believe it likely that Eliezer's CEV of humanity would not recognize your approach is better and replace CEV1 with your improved CEV2, if it is actually better? --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
Why do you believe it likely that Eliezer's CEV of humanity would not recognize your approach is better and replace CEV1 with your improved CEV2, if it is actually better? If it immediately found my approach, I would like to think that it would do so (recognize that it is better and replace Eliezer's CEV with mine). Unfortunately, it is doesn't immediately find/evaluate my approach, it might traverse some *really* bad territory while searching (with the main problem being that I perceive the proportionality attractor as being on the uphill side of the revenge attractor and Eliezer's initial CEV as being downhill of all that). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
On 03/09/2008 02:43 PM, Mark Waser wrote: Why do you believe it likely that Eliezer's CEV of humanity would not recognize your approach is better and replace CEV1 with your improved CEV2, if it is actually better? If it immediately found my approach, I would like to think that it would do so (recognize that it is better and replace Eliezer's CEV with mine). Unfortunately, it is doesn't immediately find/evaluate my approach, it might traverse some *really* bad territory while searching (with the main problem being that I perceive the proportionality attractor as being on the uphill side of the revenge attractor and Eliezer's initial CEV as being downhill of all that). It *might* get stuck in bad territory, but can you make an argument why there is a *significant* chance of that happening? Given that humanity has many times expanded the set of 'friendlies deserving friendly behavior', it seems an obvious candidate for further research. And of course, those smarter, better, more ... ones will be in a better position than us to determine that. One thing that I think most of will agree on is that if things did work as Eliezer intended, things certainly could go very wrong if it turns out that the vast majority of people -- when smarter, more the people they wish they could be, as if they grew up more together ... -- are extremely unfriendly in approximately the same way (so that their extrapolated volition is coherent and may be acted upon). Our meanderings through state space would then head into very undesirable territory. (This is the people turn out to be evil and screw it all up scenario.) Your approach suffers from a similar weakness though, since it would suffer under the seeming friendly people turn out to be evil and screw it all up before there are non-human intelligent friendlies to save us scenario. Which, if either, of 'including all of humanity' rather than just 'friendly humanity', or 'excluding non-human friendlies (initially)' do you see as the greater risk? Or is there some other aspect of Eliezer's approach that especially concerns you and motivates your alternative approach? Thanks for continuing to answer my barrage of questions. joseph --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
- Original Message - From: Matt Mahoney [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Friday, March 07, 2008 6:38 PM Subject: Re: [agi] Recap/Summary/Thesis Statement --- Mark Waser [EMAIL PROTECTED] wrote: Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) An attractor is a set of states that are repeated given enough time. NO! Easily disprovable by an obvious example. The sun (moving through space) is an attractor for the Earth and the other solar planets YET the sun and the other planets are never is the same location (state) twice (due to the movement of the entire solar system through the universe). No, the attractor is the center of the sun. The Earth and other planets are in the basin of attraction but have not yet reached equilibrium. http://en.wikipedia.org/wiki/Attractor OK. But my point is that the states of the system are NOT repeated given enough time (as you claimed and then attempted to use). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Special status to Homo Sap. (was Re: [agi] Recap/Summary/Thesis Statement)
From: Matt Mahoney [EMAIL PROTECTED], in reply to Mark Waser: You seem to be giving special status to Homo Sapiens. How does this arise out of your dynamic? I know you can program an initial bias, but how is it stable? Keep in mind that Mark made a subsequent reply saying he isn't giving a special status to Homo Sapiens. Unlike Mark, I do like to give a special status to Homo Sapiens (which I'll call humans, to save syllables). One problem with humans is that you can't prove anything about them. If you want to say your AI won't kill everbody, then you have to prove that the humans won't all be suicidal, or you have to accept that the AI might refuse to do something easy that all of the humans want. I don't like either of those options -- the net effect of this and similar scenarios is that I can't make, much less prove, any general statements about the behavior of the AI that are both interesting and useful. The only path forward here that I can see is to compare the Friendly AI with human society, rather than with some mathematical ideal. I can't prove anything about human society, but I can make plausible arguments that human society has some failure modes that the AI does not. If we expect the AI to act much more quickly than human society, we might still have a failure pretty soon just because the AI explores the state space much more quickly, so even this is bad. I don't see any hope of proving anything about the behavior of an AI that meaningfully takes human desires into account. Any ideas? Hmm, it could collect some training data about what humans want right now, and act on that forever without collecting any more training data. Right now the vast majority of the humans are not suicidal, so if the AI isn't buggy that would yield an AI that wouldn't kill everybody. Unfortunately that would limit us forever to the stupidities of the past. Right now, most humans claim to be motivated by considerations of what will happen to them in a supernatural afterlife, for example. I don't want to find out what a powerful AI would do about that. -- Tim Freeman http://www.fungible.com [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
On 03/07/2008 05:28 AM, Mark Waser wrote: */Attractor Theory of Friendliness/* There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels I've just carefully reread Eliezer's CEV http://www.singinst.org/upload/CEV.html, and I believe your basic idea is realizable in Eliezer's envisioned system. For example, if including all Friendly beings in the CEV seems preferable to our extrapolated smarter, better ... selves, then a system implementing Eliezer's approach (if working as intended) would certainly renormalize and take into account the CEV of non-humans. And if our smarter, better ... selves do not think it preferable, I'd be inclined to trust their judgment, assuming that the previous tests and confirmations that are envisioned had occurred. The CEV of humanity is only the initial dynamic, and is *intended* to be replaced with something better. joseph --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
[agi] Recap/Summary/Thesis Statement
Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels Proof: something will happen resulting in zero or more intelligent agents. Those agents will be Friendly to each other and themselves, because the action of killing agents without replacement is an irreversible dynamic, and therefore cannot be part of an attractor. Corollary: Killing with replacement is Friendly. Corollary: Friendliness does not guarantee survival of DNA based life. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels Proof: something will happen resulting in zero or more intelligent agents. Those agents will be Friendly to each other and themselves, because the action of killing agents without replacement is an irreversible dynamic, and therefore cannot be part of an attractor. Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) Corollary: Killing with replacement is Friendly. Bad Logic. Not X (replacement) leads to not Y (Friendly) does NOT have the corollary X (replacement) leads to Y (Friendliness). And I do NOT agree that Killing with replacement is Friendly. Corollary: Friendliness does not guarantee survival of DNA based life. Both not a corollary and entirely irrelevant to my points (and, in fact, in direct agreement with my statement I'm afraid that my vision of Friendliness certainly does permit the intentional destruction of the human race if that is the *only* way to preserve a hundred more intelligent, more advanced, more populous races. On the other hand, given the circumstance space that we are likely to occupy with a huge certainty, the intentional destruction of the human race is most certainly ruled out. Or, in other words, there are no infinite guarantees but we can reduce the dangers to infinitessimally small levels.) My thesis statement explicitly says acceptable levels, not guarantee. = = = = = What is your point with this e-mail? It appears to a total non-sequitor (as well as being incorrect). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: Attractor Theory of Friendliness There exists a describable, reachable, stable attractor in state space that is sufficiently Friendly to reduce the risks of AGI to acceptable levels Proof: something will happen resulting in zero or more intelligent agents. Those agents will be Friendly to each other and themselves, because the action of killing agents without replacement is an irreversible dynamic, and therefore cannot be part of an attractor. Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) An attractor is a set of states that are repeated given enough time. If agents are killed and not replaced, you can't return to the current state. Corollary: Killing with replacement is Friendly. Bad Logic. Not X (replacement) leads to not Y (Friendly) does NOT have the corollary X (replacement) leads to Y (Friendliness). And I do NOT agree that Killing with replacement is Friendly. You're right. Killing with replacement (e.g. evolution) may or may not be Friendly. Corollary: Friendliness does not guarantee survival of DNA based life. Both not a corollary and entirely irrelevant to my points (and, in fact, in direct agreement with my statement I'm afraid that my vision of Friendliness certainly does permit the intentional destruction of the human race if that is the *only* way to preserve a hundred more intelligent, more advanced, more populous races. On the other hand, given the circumstance space that we are likely to occupy with a huge certainty, the intentional destruction of the human race is most certainly ruled out. Or, in other words, there are no infinite guarantees but we can reduce the dangers to infinitessimally small levels.) My thesis statement explicitly says acceptable levels, not guarantee. You seem to be giving special status to Homo Sapiens. How does this arise out of your dynamic? I know you can program an initial bias, but how is it stable? Humans are not the pinnacle of evolution. We are a point on a curve. Is it bad that Homo Erectus is extinct? Would we be better off if they weren't? -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
--- Mark Waser [EMAIL PROTECTED] wrote: Huh? Why can't an irreversible dynamic be part of an attractor? (Not that I need it to be) An attractor is a set of states that are repeated given enough time. NO! Easily disprovable by an obvious example. The sun (moving through space) is an attractor for the Earth and the other solar planets YET the sun and the other planets are never is the same location (state) twice (due to the movement of the entire solar system through the universe). No, the attractor is the center of the sun. The Earth and other planets are in the basin of attraction but have not yet reached equilibrium. http://en.wikipedia.org/wiki/Attractor You seem to be giving special status to Homo Sapiens. How does this arise out of your dynamic? I know you can program an initial bias, but how is it stable? I am emphatically *NOT* giving special status to Homo Sapiens. In fact, that is precisely *my* objection to Eliezer's view of Friendliness. OK. That makes the problem much easier. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
