Matt Mahoney wrote:
--- "John G. Rose" <[EMAIL PROTECTED]> wrote:
Is an AGI really going to feel pain or is it just going to be some numbers?
I guess that doesn't have a simple answer. The pain has to be engineered
well for it to REALLY understand it.
An agent capable of reinforcement learning has an upper bound on the amount of
pleasure or pain it can experience in a lifetime, in an information theoretic
sense. If an agent responds to input X with output Y, followed by
reinforcement R, then we say that R is a positive reinforcement (pleasure,
R>0) if it increases the probability P(Y|X) and negative reinforcement (pain,
R<0) if it decreases P(Y|X). Let S1 be the state of the agent before R, and
S2 be the state afterwards. We may define the bound:
|R| <= K(S2|S1)
where K is Kolmogorov complexity, the length of the shortest program that
outputs an encoding of S2 given S1 as input. This definition is intuitive in
that the greater the reinforcement, the greater the change in behavior of the
agent. Also, it is consistent with the belief that higher animals (like
humans) have greater capacity to feel pleasure and pain than lower animals
(like insects) that have simpler mental states.
We must use the absolute value of R because the behavior X -> Y could be
learned using either positive reinforcement (rewarding X -> Y), negative
reinforcement (penalizing X -> not Y), or by neutral methods such as classical
conditioning (presenting X and Y together).
If you accept this definition, then an agent cannot feel more accumulated
pleasure or pain in its lifetime than K(S(death)|S(birth)). A simple program
like autobliss ( http://www.mattmahoney.net/autobliss.txt ) could not
experience more than 256 bits of reinforcement, whereas a human could
experience 10^9 bits according to cognitive models of long term memory.
I have to say that this is only one interpretation of what it would mean
for an AGI to experience something, and I for one believe it has no
validity at all. It is purely a numeric calculation that makes no
reference to what "pain" (or any other kind of subjective experience)
actually is.
Sorry, but this is such a strong point of disagreement that I have to go
on record.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=74764705-216ca2
