Hi Amir, I'm also working on how to disable the checksum, it's known issue of apache hadoop, you can find more details from the following apache JIRA : https://issues.apache.org/jira/browse/HADOOP-9114 https://issues.apache.org/jira/browse/HDFS-5761
On Mon, Apr 28, 2014 at 8:22 PM, Amir Hellershtein <[email protected]>wrote: > Hello, > > > > I wish to disable checksum calculation & verification in Hadoop's > framework, > > I run the TestDfsIO application with the parameter dfs.checksum.type=NULL, > > To my best understanding, this was supposed to turnoff check summing in > the framework, > > But I received the following error - > > > > ... > > ... > > ... > > 14/04/28 14:54:26.433 INFO hdfs.DFSOutputStream (1349) 96232.106, > 96232.10464359 : Vanilla DFSOutputStream constructed successfully. > > java.lang.ArrayIndexOutOfBoundsException: 0 > > at > org.apache.hadoop.fs.FSOutputSummer.int2byte(FSOutputSummer.java:184) > > at > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:172) > > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:148) > > at > org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:137) > > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1810) > > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66) > > at > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:99) > > at > org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:1240) > > at > org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304) > > at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:750) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:649) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > > at > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) > > at > org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:115) > > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:601) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > > 14/04/28 14:54:26.581 INFO fs.FileSystem (585) 96232.249, 0.585955522 : > Vanilla distributeFileSystem was closed. > > > > Is seems that the framework does not support disabling the checksum by the > above flag, > > Does anybody knows why I get this error? > > Is there another way to disable checksum? > > > > Thanks > > > > *Amir* > -- Best Regards, Hu Liu
