Doug Cutting wrote:
Raghu Angadi wrote:
But this will not fix the same problem with block-level checksums.
Pretty soon, HDFS will not use ChecksumFileSystem at all.
I'd hope that block-level checksums do not replicate logic from
ChecksumFileSystem. Rather they should probably share substantial
portions of their checksumming input and output stream implementations,
no? So it could fix the same problem for block-level checksums, and
should if possible.
Nope, DFSClient that implements client side of block-levels checksums
does not replicate or reuse any code from ChecksumFileSystem.
Ideally we should let the implementations decide how to buffer.
I'm not sure what you mean by this. The buffer size is a parameter to
FileSystem's open() and create() methods.
All FSOutputStreams (including DFS) go through a BufferedOutputStream.
We can not give buffersize of 0. But may be DistributedFileSystem can
always provide bufferSize of 1. I will see we can easily support
explicit option not to buffer to FSOutputStream.
Whether checksums require
another level of buffering is a separate issue.
Yes. In my patch, DFS uses FSOutputStream with default buffer so DFS is
affected by the same issue, in a different place.
Is it efficient to
invoke the CRC32 code as each byte is written, or is it faster to run it
in 512-byte or larger batches?
I think CPU cost wise, it is twice as fast to CRC32 larger buffers (>
512) than to CRC32 small buffers. But I don't think its a very
noticeable overhead. Do we expect users do many small writes?
Some measurements I did some time back:
Total size read size Total Overhead Overhead/MB MB/Overhead
---------------------------------------------------------------------
128 MB 64 1.3 sec 10.1 msec/MB 100 MB/sec
128 128 0.98 7.65 130
128 256 0.80 6.24 160
128 512 0.71 5.52 180
128 1024 0.68 5.25 190
128 10240 0.65 5.12 195
Doug
