--- On Tue, 8/26/08, Vladimir Nesov <[EMAIL PROTECTED]> wrote:
> But what is safe, and how to improve safety? This is a
> complex goal
> for complex environment, and naturally any solution to this
> goal is
> going to be very intelligent. Arbitrary intelligence is not
> safe
> (fatal, really), but what is safe is also intelligent.
Look, the bottom line is that even if you could somehow build a self-modifying
AI that was provably Friendly, some evil hacker could come along and modify the
code. One way or another, we have to treat all smarter-than-us intelligences as
inherently risky.
So safe, for me, refers instead to the process of creating the intelligence.
Can we stop it? Can we understand it? Can we limit its scope, its power? With
simulated intelligences, the answer to all of the above is yes. Pinning your
hopes of safe AI on the Friendliness of the AI is the mother of all gambles,
one that in a well-informed democratic process would surely not be undertaken.
> There is no law that makes large computations less lawful
> than small
> computations, if it is in the nature of computation to
> preserve
> certain invariants. A computation that multiplies two huge
> numbers
> isn't inherently more unpredictable than computation
> that multiplies
> two small numbers.
I'm not talking about straight-forward, linear computation. Since we're talking
about self-modification, the computation is necessarily recursive and
iterative. Recursive computation can easily lead to chaos (as in chaos theory,
not disorder).
The archetypical example of this is the simple equation from population
dynamics, y=rx(1-x), which is recursively applied for each time interval. For
values of r greater than some threshold, the behavior is chaotic and thus
unpredictable, which is a surprising result for such a simple equation.
I'm making a rather broad analogy here by comparing the above example to a
self-modifying AGI, but the principle holds. An AGI with present goal system G
computes the Friendliness of a modification M, based on G. It decides to go
ahead with the modification. This next iteration results in goal system G'. And
so on, performing Friendliness computations against the resulting goal systems.
In what sense could one guarantee that this process would not lead to chaos?
I'm not sure you could even guarantee it would continue self-modifying.
> You have intuitive
> expectation that making Z will make AI uncontrollable,
> which will lead
> to a bad outcome, and so you point out that this design
> that suggests
> doing Z will turn out bad. But the answer is that AI itself
> will check
> whether Z is expected to lead to a good outcome before
> making a
> decision to implement Z.
As has been pointed out before, by others, the goal system can drift as the
modifications are applied. The question once again is, in what *objective
sense* can the AI validate that its Friendliness algorithm corresponds to what
humans actually consider to be Friendly? What does it compare *against*?
> This remark makes my note that the field of AI actually did
> something
> for the last 50 years not that minor. Again you make an
> argument from
> ignorance: I do not know how to do it, nobody knows how to
> do it,
> therefore it can not be done. Argue from knowledge, not
> from
> ignorance. If you know the path, follow it, describe it. If
> you know
> that the path has a certain property, show it. If you know
> that a
> class of algorithms doesn't find a path, say that these
> algorithms
> won't give the answer. But if you are lost, if your map
> is blank,
> don't assert that the territory is blank also, for you
> don't know.
You can do better than that, I hope. I'm not saying it can't be done just
because I don't know how to do it. I'm giving you epistemological objections
for why Friendliness can't be specified. It's an argument from principle. If
those objections are valid, the fanciest algorithm in the world won't solve the
problem (assuming finite resources, of course). Address those objections first
before you pick on my ignorance about Friendliness algorithms.
> Causal models are not perfect, you say. But perfection is
> causal,
> physical laws are the most causal phenomenon. All the
> causal rules
> that we employ in our approximate models of environment are
> not
> strictly causal, they have exceptions. Evolution has the
> advantage of
> optimizing with the whole flow of environment, but
> evolution doesn't
> have any model of this environment, the counterpart of
> human models in
> evolution is absent. What it has is a simple regularity in
> the
> environment, natural selection. Will all the imperfections,
> human
> models of environment are immensely more precise than this
> regularity
> that relies on natural repetition of context. Evolution
> doesn't have a
> perfect model, it has an exceedingly simplistic model, so
> simple in
> fact that it managed to *emerge* by chance. Humans with
> their
> admittedly limited intelligence, on the other hand, already
> manage to
> create models far surpassing their own intelligence in
> ability to
> model the environment (computer simulations and
> mathematical models).
What I'm trying to get across here is that evolution gives us access to areas
of the solution space that our intellect can't go. Because it's possible that
the solution to AGI lies in that space, then I'm advocating putting evolution
to work and exploring that space. It's slow, it's time and resource intensive,
but those are actually advantages when you look at the risks involved with
hard-takeoff. There are other advantages as well, as I discussed in my linked
article. Admittedly, these are disadvantages when it comes to funding (slow !=
sexy), but with computational power being so cheap, it's not too expensive to
set something interesting up.
Terren
-------------------------------------------
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
https://www.listbox.com/member/?member_id=8660244&id_secret=111637683-c8fa51
Powered by Listbox: http://www.listbox.com
