--- Vladimir Nesov <[EMAIL PROTECTED]> wrote: > On Jan 28, 2008 4:53 AM, Matt Mahoney <[EMAIL PROTECTED]> wrote: > > > > Consider the following subset of possible requirements: the program is > > > correct > > > > if and only if it halts. > > > > > > > > > > It's a perfectly valid requirement, and I can write all sorts of > > > software that satisfies it. I can't take a piece of software that I > > > didn't write and tell you it it satisfies it, but I can write piece of > > > software that satisfies it, that also does all sorts of useful stuff. > > > > That is not the hard problem. Going from a formal specification (actually > a > > program) to code is just a matter of compilation. But verifying that the > > result is correct is undecidable. > > What do you mean by that? What word 'result' in your last sentence > refers to? Do you mean result of compilation? There are verified > stacks, from the ground up. Given enough effort, it should be possible > to be arbitrarily sure of their reliability. > > And anyway, what is undecidable here?
It is undecidable whether a program satisfies the requirements of a formal specification, which is the same as saying that it is undecidable whether two programs are equivalent. The halting problem reduces to it. > > Maybe AGI will solve some of these problems that seem to be beyond the > > capabilities of humans. But again it is a double edged sword. There is a > > disturbing trend in attacks. Attackers used to be motivated by ego, so > you > > had viruses that played jokes or wiped your files. Now they are motivated > by > > greed, so attacks remain hidden while stealing personal information and > > computing resources. Acquiring resources is the fitness function for > > competing, recursively self improving AGI, so it is sure to play a role. > > Now THAT you can't oppose, competition for resources by deception that > relies on human gullibility. But it's a completely different problem, > it's not about computer security at all. It's about human phychology, > and one can't do anything about it, as long as they remain human. It > probably can be kind of solved by placing generally intelligent > 'personal firewalls' on all input that human receives. The problem is not human gullibility but human cognitive limits in dealing with computer complexity. Twenty years ago ID theft, phishing, botnets, and spyware were barely a problem. This problem will only get worse as software gets more complex. What you are suggesting is to abdicate responsibility to the software, pitting ever smarter security against ever smarter intruders. This only guarantees that when your computer is hacked, you will never know. But I fear this result is inevitable. Here is an example of cognitive load. Firefox will pop up a warning if you visit a known phishing site, but this doesn't work every time. It also makes such sites easier to detect because when you hover the mouse over a link, it shows the true URL because by default Firefox disables Javascript code that hackers add to write a fake URL to the status bar (which is enabled in IE and can be enabled in Firefox). This is not foolproof against creative attacks such as registering www.paypaI.com (with a capitol I) or attacking routers or DNS servers to redirect traffic to bogus sites, or sniffing traffic to legitimate sites, or keyboard loggers capturing your passwords, or taking advantage of users who use the same password on more than one site to reduce their cognitive load (something you would never do, right?) I use Firefox because I think it is more secure than IE, even though there seems to be a new attack discovered about once a week. http://www.mozilla.org/projects/security/known-vulnerabilities.html Do you really expect users to keep up with this, plus all their other software? No. You will rely on AGI to do it for you, and when it fails you will never know. -- Matt Mahoney, [EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=90580840-9cbff8
