[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845510#comment-13845510
]
Jimmy Xiang commented on HBASE-9870:
There is no solid proof that this is an actually issue causing data loss in
IntegrationTestBigLinkedList if online encoding change is enabled. It is an
improvement to make the data encoding in cache the same as that on disk, to
simplify the code and avoid confusion. For now, I think we may not need this
in 0.94.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845069#comment-13845069
]
Lars Hofhansl commented on HBASE-9870:
--
[~jxiang], Don't we need the same fix in 0.94?
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[ https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822552#comment-13822552 ] Hudson commented on HBASE-9870: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format (jxiang: rev 1541629) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java *
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822667#comment-13822667
]
Hudson commented on HBASE-9870:
---
SUCCESS: Integrated in hbase-0.96 #189 (See
[https://builds.apache.org/job/hbase-0.96/189/])
HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
(jxiang: rev 1541627)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/branches/0.96/hbase-shell/src/main/ruby/hbase/admin.rb
* /hbase/branches/0.96/src/main/docbkx/shell.xml
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
if
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[ https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822888#comment-13822888 ] Hudson commented on HBASE-9870: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format (jxiang: rev 1541629) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java *
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822912#comment-13822912
]
Hudson commented on HBASE-9870:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/119/])
HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
(jxiang: rev 1541627)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/branches/0.96/hbase-shell/src/main/ruby/hbase/admin.rb
* /hbase/branches/0.96/src/main/docbkx/shell.xml
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820384#comment-13820384
]
Jimmy Xiang commented on HBASE-9870:
One more thing, the RegionObserver methods we touched are not in 0.96. So we
can change the interface, no need to deprecate.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820391#comment-13820391
]
Hadoop QA commented on HBASE-9870:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613399/trunk-9870_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 54 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7819//console
This message is automatically generated.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820923#comment-13820923
]
Hadoop QA commented on HBASE-9870:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613461/trunk-9870_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 57 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7829//console
This message is automatically generated.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819157#comment-13819157
]
Jimmy Xiang commented on HBASE-9870:
Posted the patch on RB: https://reviews.apache.org/r/15421/
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819162#comment-13819162
]
Jimmy Xiang commented on HBASE-9870:
The patch makes sure the data block encoding in cache is always the same as
that on disk. With the patch (+HBASE-9926), ITBLL is green now with aggressive
region movements and online encoding change.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819252#comment-13819252
]
Hadoop QA commented on HBASE-9870:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613185/trunk-9870.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 54 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7812//console
This message is automatically generated.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13814420#comment-13814420
]
Jimmy Xiang commented on HBASE-9870:
In BlockCacheKey, we do have the encoding format. However, the equal method
doesn't check the encoding format, which may be interesting.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812421#comment-13812421
]
Jimmy Xiang commented on HBASE-9870:
I tried to make sure onDisk encoding always the same as the inCache one, which
still gave me data loss. Perhaps I missed something. I will get rid of the
inCache one and give it another try.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812028#comment-13812028
]
Andrew Purtell commented on HBASE-9870:
---
I agree with Matt. Let's just load disk contents to cache, with a warning, if
the settings are different, that the in-cache setting is being ignored. And
deprecate the in-cache setting constants and methods.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812254#comment-13812254
]
stack commented on HBASE-9870:
--
[~jxiang] I wonder if you did as @matt corgan suggested, does the dataloss
issue go away (i.e. you'd kill two birds w/ one stone -- a simplification and
get rid of your bane)
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811072#comment-13811072
]
Anoop Sam John commented on HBASE-9870:
---
Yes this is to avoid a decode and then encode in the read path which is costly.
At any point of time there can be only same type of encoding for OnDisk and
InCache for a CF. (If OnDisk != None)
But when we alter the encoding for a CF!!! Yes at time also both in cache and
on disk are same.. But the existing data!!! On disk that can be in another
encoding format.. So this is an issue.
Very good finding Jimmy..
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811081#comment-13811081
]
stack commented on HBASE-9870:
--
[~anoop.hbase] for my understanding, when you say 'the existing data', do you
mean data that has already been loaded into the block cache or do you mean
something else? (Changing the encoding, you have to offline the region?). Or
is 'existing data' something else? Won't the data in hfiles be properly
converted from old encoding to new? Thanks.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811481#comment-13811481
]
Nick Dimiduk commented on HBASE-9870:
-
[~jxiang] can you describe how to reproduce the data loss you're seeing? Maybe
a (integration) test case?
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811491#comment-13811491
]
Jimmy Xiang commented on HBASE-9870:
[~ndimiduk], apply the patches of 9792 and 9757 to trunk, then run ITBLL, it
will fail with data loss. After you comment out new
ChangeEncodingAction(tableName), in SlowDeterministicMonkeyFactory, you run
ITBLL again, it will pass.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811492#comment-13811492
]
Anoop Sam John commented on HBASE-9870:
---
I mean the data already there in HFiles (on disk). That will be with old
encoding algo. Now the new encoding algo for on disk and in cache is same but
different from this old one. When we read data from old HFiles and have to load
to block cache we need a decoding 1st and then encoding which we are not at all
doing now. We expect the either the HFile data is with no encoding or with
same encoding as what block cache expecting.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811523#comment-13811523
]
Matt Corgan commented on HBASE-9870:
I forget the intricacies (or never understood them). I think the facebook guys
added this ability to have different on-disk/in-cache encodings because they
wanted to slowly test encoding without changing/corrupting the on-disk format,
but now this safety feature has become dangerous. In my opinion, we should
just remove that feature and always load from on-disk to in-cache without
changing the format. To change the encoding, you would run a compaction.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811646#comment-13811646
]
Jimmy Xiang commented on HBASE-9870:
[~anoop.hbase], when we open a hfile, do we also use the encoding from the file?
{code}
// Read data block encoding algorithm name from file info.
dataBlockEncoder = HFileDataBlockEncoderImpl.createFromFileInfo(fileInfo,
preferredEncodingInCache);
fsBlockReaderV2.setDataBlockEncoder(dataBlockEncoder);
{code}
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811863#comment-13811863
]
Anoop Sam John commented on HBASE-9870:
---
Yes when we open a HFile, we will use the encoding stored in FileInfo. Down
the line when we read blocks of data there will be the check that you mentioned
in the issue right? That we dont want different encoding scheme for onDisk and
inCache (onDisk != NONE)
In case when we are altering a CF with changing encoding from one alogo to
another this situation arises. My feeling is that is not been handled. So
this situation requires data encoded in a file using one algo should get
encoded using another algo and get stored in cache.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810789#comment-13810789
]
Jimmy Xiang commented on HBASE-9870:
As to the fix, we should do a decoding from onDisk, then an encoding to
inCache, if onDisk != inCache.
What do you think, [~mcorgan]?
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810811#comment-13810811
]
Jimmy Xiang commented on HBASE-9870:
It seems we always make sure onDisk == inCache if onDisk != NONE. So this is
not an issue at all?
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811048#comment-13811048
]
ramkrishna.s.vasudevan commented on HBASE-9870:
---
[~jxiang]
Ya we always make sure onDisk==inCache. There can not be two different algos.
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
In this method, we have
{code}
if (block.getBlockType() == BlockType.ENCODED_DATA) {
if (block.getDataBlockEncodingId() == onDisk.getId()) {
// The block is already in the desired in-cache encoding.
return block;
}
{code}
This assumes onDisk encoding is the same as that of inCache. This is not
true when we change the encoding of a CF. This could be one of the reasons I
got data loss with online encoding change?
If I make sure onDisk == inCache all the time, my ITBLL with online encoding
change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
