sodonnel commented on PR #5364: URL: https://github.com/apache/ozone/pull/5364#issuecomment-1736354885
@jojochuang You mentioned you still noticed some overhead on a NVME cluster - do you recall what that was? I guess the checksum calculation itself would still have overhead, but this "hack" you mentioned would avoid the memory allocation and an extra copy, so it makes the checksum calculation as fast as the algorithm allows. I guess the write rate onto NVME may be so high the checksum overhead is significant. In my benchmarks, I was getting about 2000 ops per section on a single thread calculating 1M bytes per checksum on 4MB of data, which is about 8000MB/s per thread. The hack you suggested may not be perfect, but its a lot better than what we have currently with the extra copy! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
