Sorry, its not a problem (lib path mistake in my environment) Thanks -Shankar
[X] This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! [X] _____________________________________________ From: Shankar hiremath Sent: 01 August 2014 PM 07:53 To: '[email protected]' Subject: RE: Problem reading HFile Trailer by using HFile tool Actually the AES encryption for Hfile & WAL is enabled in hbase-site.xml. But for the table 't3' column Family 'cf1' is not encrypted (because during table creation we did not specify ENCRYPTION => 'AES') So basically the 'cf1/5817635667d7457989b6d0b0be25dbc4' is compressed with SNAPPY (no AES encryption, on this Hfile) In this case ./hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4 Is getting failed with 'Problem reading HFile Trailer by using HFile tool' Thanks -Shankar [X] This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! [X] _____________________________________________ From: Shankar hiremath Sent: 01 August 2014 PM 07:43 To: [email protected]<mailto:[email protected]> Subject: Problem reading HFile Trailer by using HFile tool When I am reading the Hfile using the './hbase org.apache.hadoop.hbase.io.hfile.Hfile' tool (the HFile which is compressed with SNAPPY & encrypted with AES) I am getting the below error "Problem reading HFile Trailer by using HFile tool" Is there any problem in the usage of the below command, or is it a bug. ------------------------------------------------------------------------------------------ Version details: Hadoop 2.4.1, HBase 0.98.3 Configuration Details: (enabled Hfile and Wal encryption AES, as below) hfile.format.version=3 hbase.crypto.keyprovider=org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider hbase.crypto.keyprovider.parameters= jceks:///opt/shankar1/kdc_keytab/hbase.jks?password=shankar@234 We created a table with SNAPPY compression > Create 't3', {NAME => 'cf1', COMPRESSION => 'SNAPPY'} > Put 't3','r1','cf1:a','1000' > Flush 't3' shankar1@host1:~/DataSight/hbase/bin> ./hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4 2014-08-01 19:18:28,368 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 2014-08-01 19:18:28,504 INFO [main] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32 2014-08-01 19:18:28,506 INFO [main] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C 2014-08-01 19:18:28,739 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-01 19:18:29,082 INFO [main] hdfs.DFSClient: Set dfs.client.block.write.replace-datanode-on-failure.replication to 0 2014-08-01 19:18:29,406 INFO [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS Scanning -> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4 2014-08-01 19:18:29,409 INFO [main] hdfs.DFSClient: Set dfs.client.block.write.replace-datanode-on-failure.replication to 0 INFO: Watching file:/opt/shankar1/DataSight/hbase/conf/log4j.properties for changes with interval : 60000 2014-08-01 19:18:29,779 ERROR [main] hfile.HFilePrettyPrinter: Error reading hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4 at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:595) at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217) at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196) at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:873) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:190) at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176) at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:336) at org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:433) at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:91) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1522) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1151) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1159) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:146) at org.apache.hadoop.hbase.io.hfile.HFileReaderV3.<init>(HFileReaderV3.java:72) at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:542) ... 4 more Scanned kv count -> 0 shankar1@host1:~/DataSight/hbase/bin> Thanks -Shankar [X] This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! [X]
