Raghu Angadi wrote:
In my implementation of block-level CRCs (does not affect ChecksumFileSystem in HADOOP-928), we don't buffer checksum data at all.
That sounds like a good approach. I look forward to seeing the patch.
We could remove buffering all together in FileSystem level and let the FS implementations to decide how to buffer.
That's already been done, as of HADOOP-928. FileSystem implementations now opt to use ChecksumFileSystem. The buffer size defaults to io.buffer.size, but applications may pass an explicit buffer size to the FileSystem. The FileSystem implementation is free to ignore that hint.
Doug
