On Apr 14, 2011, at 9:19 PM, Michael_google gmail_Gersten wrote: >> The question is, can data be corrupted without detection or correction, over >> TCP/IP? It seems like the checksumming used by TCP/IP is considered weak by >> modern standards. So there is the possibility of undetected and uncorrected >> corruption, but I don't know how probable it is. > > Answer: Yes. > > Even if you assume that TCP gives you 100% network transmission > accuracy, that does NOTHING for driver copy error, or memory failure, > etc.
I understand, I'd like to just leave this at the physical layer at the moment. I'm curious if I understand TCP correctly, that just the header is checksummed, not the data. > > Look up silent data corruption. Be amazed that you can get double bit > uncorrectable errors from systems where that is supposed to be > impossible. Be amazed that some hardware will, under reproducible > situations, give you 100% chance of a corrupt packet with no hardware > notice that it is corrupt. I'm not surprised at all I've seen lots of data on this for hard drives and controllers, and as it turns out a lot of silent data corruption is due to bugs in the firmware actually CAUSING the corruption in the first place. So really without ZFS or btrfs (or other file system capable of vastly superior checksumming) our filesystems leave us no mechanism to verify the veracity of the data on the drive. We simply trust that the drive has read the sector correctly with its checksum and has detected/corrected before providing it to the file system - and that it hasn't been fakaked in between the drive and RAM or in RAM. Drive checksumming is getting better with Advanced Format drives. But still, what I'm most curious about is the TCP aspect as I have not been able to find much information on this. And how to mitigate it either with detection or better detection and correction (ala ZFS or btrfs). So I wonder if encryption has some mechanisms here, like IPSec. It seems like any amount of corruption that's not detected (and corrected) would completely pollute the data stream as it wouldn't be decrypted correctly. > > You have to have end to end, final check of transmitted data. > Lacking this, you have at best a mostly error free, unvalidated stream. > > Yes, since IP checksums and TCP checksums are calculated in the > driver, any error in the packet going in/out of the network hardware > should be detected. That still means at least one copy, possibly more, > in memory without error checking. Again, I'll have to pass on the memory factor because there's usually no ECC memory except in servers, not much I can do about this and I'll just have to accept the chances a cosmic ray corrupts something periodically. But I wonder if something can be done about corruption detection and correction occurring in the physical layer - assuming that TCP is presently only checksumming headers, not data. Or if data also, some estimate of how adequate it is. It seems a little ridiculous to use IPSec in an internal network just to ensure the valid transport of data, but maybe this is actually done for that very reason and not just for security purposes? Chris Murphy_______________________________________________ MacOSX-admin mailing list [email protected] http://www.omnigroup.com/mailman/listinfo/macosx-admin
