shahrs87 commented on a change in pull request #3244:
URL: https://github.com/apache/hbase/pull/3244#discussion_r630264621
##########
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
##########
@@ -256,6 +267,43 @@ public void write(Cell cell) throws IOException {
}
}
}
+
+ private byte[] compressValue(Cell cell) throws IOException {
+ Deflater deflater = compression.getValueCompressor().getDeflater();
+ if (cell instanceof ByteBufferExtendedCell) {
+
deflater.setInput(((ByteBufferExtendedCell)cell).getValueByteBuffer().array(),
+ ((ByteBufferExtendedCell)cell).getValueByteBuffer().arrayOffset() +
+ ((ByteBufferExtendedCell)cell).getValuePosition(),
+ cell.getValueLength());
+ } else {
+ deflater.setInput(cell.getValueArray(), cell.getValueOffset(),
cell.getValueLength());
+ }
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review comment:
@apurtell I was trying to understand how this buffer works while we
compress the data. So I wrote a small test case to understand it better.
```
@Test
public void testCodec() throws Exception {
String inputRawString = "helloworld";
int rawStringLength = inputRawString.length();
byte[] compressedBytesArr = compressValue(inputRawString);
int compressedBytesLength = compressedBytesArr.length;
}
private byte[] compressValue(String inputRawString) throws IOException {
Deflater deflater = new Deflater();
deflater.setLevel(Deflater.BEST_SPEED);
byte[] input = inputRawString.getBytes("UTF-8");
deflater.setInput(input, 0, input.length);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[5];
int bytesOut;
do {
bytesOut = deflater.deflate(buffer);
if (bytesOut > 0) {
baos.write(buffer, 0, bytesOut);
}
} while (bytesOut > 0);
// Done compressing value, now flush until deflater buffers are empty
do {
bytesOut = deflater.deflate(buffer, 0, buffer.length,
Deflater.SYNC_FLUSH);
baos.write(buffer, 0, bytesOut);
} while (bytesOut == buffer.length); // See javadoc for Deflater#deflate
return baos.toByteArray();
}
```
Basically the above snippet is copied pasted from your PR where you compress
the cell. I replcaed the cell with some random String.
Also I made sure that the size of buffer (5) is less than input string
length (which is 10) since in real world the size of Cell value will be more
than 1024 which is the buffer size in PR.
The above test goes into indefinite while loop in the second do..while loop.
If I change the size of buffer to 10 then it succeeds.
Please educate me if I am doing something wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
