> > It should be even enough to take the first and last 512K of a message. > With intelligent hardware and zero-copy mechanisms, it could be > possible to directly pass the data stream into SQL, without copying it > around in memory. Then the checksum calculation can be a big > performance hit. Also, by taking the last n bytes of a messages, file > formats like AVI that have the header in the end get a better checksum, > whenever the beginning of a video is the same and it just differs > later. > > There is no real reason not to sha/md5 the whole thing. My desktop 2.6Ghz Pentium-D uses about 25% of one core while processing 50Mbytes per second. If you are getting 50MBytes per second you can afford an extra core to spend on the hashing and pay for it with the money saved on hard drives ($160 for a 320Gb hdd every hour and a half or so)
_______________________________________________ DBmail mailing list [email protected] https://mailman.fastxs.nl/mailman/listinfo/dbmail
