linliu-code commented on code in PR #13873:
URL: https://github.com/apache/hudi/pull/13873#discussion_r2335907278
##########
hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileDataBlock.java:
##########
@@ -203,17 +205,30 @@ protected ByteBuffer getUncompressedBlockDataToWrite() {
ByteBuffer dataBuf = ByteBuffer.allocate(context.getBlockSize());
for (KeyValueEntry kv : entriesToWrite) {
// Length of key + length of a short variable indicating length of key.
- dataBuf.putInt(kv.key.length + SIZEOF_INT16);
+ // Note that 10 extra bytes are required by hbase reader.
+ // That is: 1 byte for column family length, 8 bytes for timestamp, 1
bytes for key type.
+ dataBuf.putInt(kv.key.length + SIZEOF_INT16 + SIZEOF_BYTE + SIZEOF_INT64
+ SIZEOF_BYTE);
// Length of value.
dataBuf.putInt(kv.value.length);
// Key content length.
dataBuf.putShort((short)kv.key.length);
// Key.
dataBuf.put(kv.key);
+ // Column family length: constant 0.
+ dataBuf.put((byte)0);
+ // Column qualifier: assume 0 bits.
+ // Timestamp: constant 0.
+ dataBuf.putLong(0L);
+ // Key type: constant Put (4) in Hudi.
+ // Minimum((byte) 0), Put((byte) 4), Delete((byte) 8),
+ // DeleteFamilyVersion((byte) 10), DeleteColumn((byte) 12),
+ // DeleteFamily((byte) 14), Maximum((byte) 255).
+ dataBuf.put((byte)4);
Review Comment:
Because these fields are not needed in Hudi, therefore in our tests, no
matter we set them or not, they are skipped. However in Hbase hfile reader, all
fields are verified to see if their format is valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
