No serialization framework that I know of does checksums as a matter of course.
All large scale computing frameworks that I know of do them. The best ones do the checksum at the client file system layer before even sending requests to the server. Bit errors definitely exist, but I saw more errors on disk 10 years ago than have lately even with the dramatic increases in size. The failure rate for drives is somewhat less, but still about the same but the transient error rate is down dramatically on a per byte stored basis. On Mon, May 30, 2011 at 2:59 PM, Lance Norskog <[email protected]> wrote: > Do any of these encapsulators do checksums? > > In light of the recent Google paper about rampant bit errors, and > personal experience with high-end disk arrays, it is clear to me that > that all stored data should have checksums. File systems, app-level > encapsulations, the works. (TCP/IP has checksums, but they are quite > weak.) > > On Mon, May 30, 2011 at 12:02 PM, Ted Dunning <[email protected]> > wrote: > > Kryo is java specific. > > > > I would really like to use a portable solution. > > > > On Mon, May 30, 2011 at 6:02 AM, Grant Ingersoll <[email protected] > >wrote: > > > >> Seems to me like kyro and protostuff (not protobuff) are the big winners > in > >> that one. Avro's competitive on size, but not time. Granted, one > should > >> never read too much into benchmarks, but... > >> > > > > > > -- > Lance Norskog > [email protected] >
