I had the same thing happen to me and it was because I had bad RAM in my computer. Once I tested the RAM one stick at a time and found out which stick was the bad one and took it out of my computer, OpenSolaris installed fine and was rock solid.
A few months after installing it, I tried to upgrade my RAM and accidentally put yet another bad stick in (without knowing it) and my computer started spitting out garbage on the screen every time I did a ZFS zpool scrub and it would also reboot randomly when I started running lots of virtual machines in virtual box. Bad RAM is everywhere and it's the silent killer, you could be running Windows Server 2003 and have an e-commerce site and you won't know it's in there until your site becomes popular and gets a lot of hits and suddenly the OS is actually using up a lot of RAM causing it to hit the bad spot on the bad RAM and reboot all the time. I've noticed that OpenSolaris tends to use up parts of RAM memory that aren't being used by other applications for ZFS related things, and this makes it find the bad spots on bad RAM more quickly than Windows or Linux or BSD or other operating systems do. Please post what kind of RAM you're using, where you got it from, and how much of it is in your computer. -- This message posted from opensolaris.org