[
https://issues.apache.org/jira/browse/HBASE-27637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17688252#comment-17688252
]
Duo Zhang commented on HBASE-27637:
-----------------------------------
I modified CompressedWALTestBase to include a deleteFamily cell so I can not
reproduce the problem. And I found a typo in our current
TestCompressedWALValueCompression, as we declare a HBaseTestingUtil in the sub
class, so setting the configuration of this HBTU does not affect the one in the
parent class, so actual we just test no compress...
After fixing this simple typo, I could reproduce the problem, if we add a
deleteFamily cell where its value is null, the read will stop here, and cause
the test to fail like this
{noformat}
java.lang.AssertionError: Should have read back as many KVs as written
expected:<1000> but was:<1>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at
org.apache.hadoop.hbase.wal.CompressedWALTestBase.doTest(CompressedWALTestBase.java:123)
at
org.apache.hadoop.hbase.wal.TestCompressedWALValueCompression.test(TestCompressedWALValueCompression.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
{noformat}
> Zero length value would cause value compressor read nothing and not advance
> the position of the InputStream
> -----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-27637
> URL: https://issues.apache.org/jira/browse/HBASE-27637
> Project: HBase
> Issue Type: Bug
> Components: dataloss, wal
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Critical
>
> This is a code sniff from the discussion of HBASE-27073
> {code}
> public static void main(String[] args) throws Exception {
> CompressionContext ctx =
> new CompressionContext(LRUDictionary.class, false, false, true,
> Compression.Algorithm.GZ);
> ValueCompressor compressor = ctx.getValueCompressor();
> byte[] compressed = compressor.compress(new byte[0], 0, 0);
> System.out.println("compressed length: " + compressed.length);
> ByteArrayInputStream bis = new ByteArrayInputStream(compressed);
> int read = compressor.decompress(bis, compressed.length, new byte[0], 0,
> 0);
> System.out.println("read length: " + read);
> System.out.println("position: " + (compressed.length - bis.available()));
> {code}
> And the output is
> {noformat}
> compressed length: 20
> read length: 0
> position: 0
> {noformat}
> So it turns out that, when compressing, an empty array will still generate
> some output bytes but while reading, we will skip reading anything if we find
> the output length is zero, so next time when we read from the stream, we will
> start at a wrong position...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
