Ben, you and I have a long-standing disagreement on a certain issue which
impacts the survival of all life on Earth. I know you're probably bored
with it by now, but I hope you can understand why, given my views, I keep
returning to it, and find a little tolerance for my doing so.
The issue
I can spot the problem in AIXI because I have practice looking for silent
failures, because I have an underlying theory that makes it immediately
obvious which useful properties are formally missing from AIXI, and
because I have a specific fleshed-out idea for how to create
moral systems
Your intuitions say... I am trying to summarize my impression of your
viewpoint, please feel free to correct me... AI morality is a matter of
experiential learning, not just for the AI, but for the programmers.
Also, we plan to start Novamente off with some initial goals embodying
ethical
Hi,
2) If you get the deep theory wrong, there is a strong possibility of a
silent catastrophic failure: the AI appears to be learning
everything just
fine, and both you and the AI are apparently making all kinds of
fascinating discoveries about AI morality, and everything seems to be
Eliezer S. Yudkowsky wrote:
1) AI morality is an extremely deep and nonobvious challenge which has
no significant probability of going right by accident.
2) If you get the deep theory wrong, there is a strong possibility of
a silent catastrophic failure: the AI appears to be learning
Ben Goertzel wrote:
Your intuitions say... I am trying to summarize my impression of your
viewpoint, please feel free to correct me... AI morality is a
matter of experiential learning, not just for the AI, but for the
programmers. To teach an AI morality you must give it the right
feedback
This is slightly off-topic but no more so than the rest of the thread...
1) That it is selfishly pragmatic for a superintelligence to deal with
humans economically rather than converting them to computronium.
For convenience, lets rephrase this
the majority of arbitrarily generated
Jonathan Standley wrote:
Now here is my question, it's going to sound silly but there is
quite a bit behind it:
Of what use is computronium to a superintelligence?
If the superintelligence perceives a need for vast computational
resources, then computronium would indeed be very useful.
Eliezer S. Yudkowsky wrote:
I recently read through Marcus
Hutter's AIXI paper, and while Marcus Hutter has done valuable work on a
formal definition of intelligence, it is not a solution of Friendliness
(nor do I have any reason to believe Marcus Hutter intended it as one).
In fact, as one
Eliezer wrote:
* a paper by Marcus Hutter giving a Solomonoff induction based theory
of general intelligence
Interesting you should mention that. I recently read through Marcus
Hutter's AIXI paper, and while Marcus Hutter has done valuable work on a
formal definition of intelligence,
In a message dated 2/11/2003 10:17:07 AM Mountain Standard Time, [EMAIL PROTECTED] writes:
1) There is a class of physically realizable problems, which humans can
solve easily for maximum reward, but which - as far as I can tell - AIXI
cannot solve even in principle;
2) While an AIXI-tl of
2) While an AIXI-tl of limited physical and cognitive capabilities
might serve as a useful tool, AIXI is unFriendly and cannot be made
Friendly regardless of *any* pattern of reinforcement delivered during
childhood.
Before I post further, is there *anyone* who sees this besides me?
On Tue, 11 Feb 2003, Ben Goertzel wrote:
Eliezer wrote:
* a paper by Marcus Hutter giving a Solomonoff induction based theory
of general intelligence
Interesting you should mention that. I recently read through Marcus
Hutter's AIXI paper, and while Marcus Hutter has done valuable
-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On
Behalf Of Ben Goertzel
Sent: Tuesday, February 11, 2003 4:33 PM
To: [EMAIL PROTECTED]
Subject: RE: [agi] unFriendly AIXI
The formality of Hutter's definitions can give the impression
that they cannot evolve. But they are open
Ben Goertzel wrote:
AIXI and AIXItl are systems that are designed to operate with an initial
fixed goal. As defined, they don't modify the overall goal they try to
achieve, they just try to achieve this fixed goal as well as possible
through adaptively determining their actions.
Basically, at
Ben,
On Tue, 11 Feb 2003, Ben Goertzel wrote:
The formality of Hutter's definitions can give the impression
that they cannot evolve. But they are open to interactions
with the external environment, and can be influenced by it
(including evolving in response to it). If the reinforcement
The harmfulness or benevolence of an AIXI system is therefore
closely tied
to the definition of the goal that is given to the system in advance.
Actually, Ben, AIXI and AIXI-tl are both formal systems; there is no
internal component in that formal system corresponding to a goal
Ben Goertzel wrote:
The harmfulness or benevolence of an AIXI system is therefore closely tied
to the definition of the goal that is given to the system in advance.
Under AIXI the goal is not given to the system in advance; rather, the
system learns the humans' goal pattern through Solomonoff
Ben Goertzel wrote:
Huh. We may not be on the same page. Using:
http://www.idsia.ch/~marcus/ai/aixigentle.pdf
Page 5:
The general framework for AI might be viewed as the design and study of
intelligent agents [RN95]. An agent is a cybernetic system with some
internal state, which acts with
Given
this, would you regard AIXI as formally approximating the kind of goal
learning that Novamente is supposed to do?
Sorta.. but goal-learning is not the complete motivational structure of
Novamente... just one aspect
As Definition 10 makes clear, intelligence is defined relative
to
Ben Goertzel wrote:
Yeah, you're right, I mis-spoke. The theorems assume the goal function is
known in advance -- but not known to the system, just known to the entity
defining and estimating the system's intelligence and giving the rewards.
I was implicitly assuming the case in which the goal
Eliezer S. Yudkowsky wrote:
Not really. There is certainly a significant similarity between Hutter's
stuff and the foundations of Novamente, but there are significant
differences too. To sort out the exact relationship would take me
more than a few minutes' thought.
There are indeed major
Eliezer,
In this discussion you have just moved the focus to the superiority of
one AGI approach versus another in terms of *interacting with
humans*.
But once one AGI exists it's most likely not long before there are more
AGIs and there will need to be a moral/ethical system to guide AGI-AGI
Hi,
The reason I asked the question was not to ask whether AIXI is
pragmatically better as a design strategy than Novamente. What I was
asking you rather is if, looking at AIXI, you see something
*missing* that
would be present in Novamente. In other words, *if* you had an
infinitely
Bill Hibbard wrote:
On Tue, 11 Feb 2003, Ben Goertzel wrote:
Eliezer wrote:
Interesting you should mention that. I recently read through Marcus
Hutter's AIXI paper, and while Marcus Hutter has done valuable work on a
formal definition of intelligence, it is not a solution of Friendliness
Oh, well, in that case, I'll make my statement more formal:
There exists a physically realizable, humanly understandable challenge C
on which a tl-bounded human outperforms AIXI-tl for humanly
understandable
reasons. Or even more formally, there exists a computable process P
which, given
Ben Goertzel wrote:
Oh, well, in that case, I'll make my statement more formal:
There exists a physically realizable, humanly understandable
challenge C on which a tl-bounded human outperforms AIXI-tl for
humanly understandable reasons. Or even more formally, there exists
a computable
So what clever loophole are you invoking?? ;-)
An intuitively fair, physically realizable challenge with important
real-world analogues, solvable by the use of rational cognitive reasoning
inaccessible to AIXI-tl, with success strictly defined by reward (not a
Friendliness-related issue).
It seems to me that this answer *assumes* that Hutter's work is completely
right, an assumption in conflict with the uneasiness you express in your
previous email.
It's right as mathematics...
I don't think his definition of intelligence is the maximally useful one,
though I think it's a
Ben Goertzel wrote:
It's right as mathematics...
I don't think his definition of intelligence is the maximally useful
one, though I think it's a reasonably OK one.
I have proposed a different but related definition of intelligence,
before, and have not been entirely satisfied with my own
30 matches
Mail list logo
