Fellow HDFS developers, Want to bring this issue up to your attention: HDFS-10935 <https://issues.apache.org/jira/browse/HDFS-10935> (Java-based EC codec does not reconstruct blocks correctly)
Initially I found 9 out of 30 TestFileChecksum tests failed consistently on my Mac IntelliJ. More specifically, they failed with error “Checksum mismatch”. Because jenkins job never picked up this failure, I thought it must have been some kind of environment issue. However, after consulting a number of fellow HDFS developers (Kai, Sammi, Eddy, Sean, Andrew and John), it turns out that this set of tests fail without native ISA-L library. I was able to set up two local Hadoop repo on a Linux box, one is built with native ISA-L lib and the other isn’t. And the one with native lib always succeeds and the other always fails. If I understand them correctly, these tests verifies that blocks are reconstructed correctly after one datanode is shutdown, and therefore, the checksum mismatch w/o native ISA-L lib seems to suggest that Java-based EC codec does not reconstruct strips correctly. I do not have sufficient insight into EC reconstruction so I maybe wrong to interpret the failures this way. But I think this is severe enough to become a blocker for the next Hadoop 3 alpha release. Best, Wei-Chiu Chuang A very happy Clouderan
