On 5/4/05 19:46:56, Terry Creer wrote: > >>> Turns out to be a total of roughly 350 inches (!) > >> > >> Yes. No wonder there was overflow. I think the basic internal database > >> unit is the microinch, and I would guess that it is a binary > >> representation. > > > > I didn't quite follow here.. Even if a microinch was the internal unit, a > > standard 4 byte unsigned integer would be able to hold 4.3e9 microinches, > > or 4300 inches. Close, but not quite there. And even if they had used > > signed integer (now why would that be?), it's still over 2000 inches.
> That's assuming that one bit is one mil, yes? > I've seen Protel measure mils up to 3 or 4 decimal places. A mil is a thousands of an inch. A microinch is 1e-6 of an inch. (For more info about standard prefixes see http://physics.nist.gov/Pubs/SP811/sec04.html). So if Protel works with 1/10 of a microinch, a 4 byte unsigned int would still hold 430 inches. Maybe they use signed int... you never know. In any case, this would be rather close to an overflow situation. Gerhard ____________________________________________________________ You are subscribed to the PEDA discussion forum To Post messages: mailto:[email protected] Unsubscribe and Other Options: http://techservinc.com/mailman/listinfo/peda_techservinc.com Browse or Search Old Archives (2001-2004): http://www.mail-archive.com/[email protected] Browse or Search Current Archives (2004-Current): http://www.mail-archive.com/[email protected]
