yihua commented on code in PR #13873:
URL: https://github.com/apache/hudi/pull/13873#discussion_r2338324910
##########
hudi-io/src/test/java/org/apache/hudi/io/hfile/TestHFileWriter.java:
##########
@@ -72,7 +75,7 @@ void testOverflow() throws Exception {
@Test
void testSameKeyLocation() throws IOException {
// 50 bytes for data part limit.
- HFileContext context = new HFileContext.Builder().blockSize(50).build();
+ HFileContext context = new HFileContext.Builder().blockSize(100).build();
Review Comment:
It looks like the test fails because of the byte buffer allocation in
`HFileFileInfoBlock#getUncompressedBlockDataToWrite`. That needs to be fixed.
##########
hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileContext.java:
##########
@@ -31,12 +31,17 @@ public class HFileContext {
private final HoodieCompressor compressor;
private final ChecksumType checksumType;
private final int blockSize;
+ private final long fileCreateTime;
- private HFileContext(CompressionCodec compressionCodec, int blockSize,
ChecksumType checksumType) {
+ private HFileContext(CompressionCodec compressionCodec,
+ int blockSize,
+ ChecksumType checksumType,
+ long fileCreateTime) {
this.compressionCodec = compressionCodec;
this.compressor = HoodieCompressorFactory.getCompressor(compressionCodec);
this.blockSize = blockSize;
this.checksumType = checksumType;
+ this.fileCreateTime = fileCreateTime;
Review Comment:
Fixed.
##########
hudi-io/src/main/java/org/apache/hudi/io/hfile/HFileInfo.java:
##########
@@ -31,32 +31,31 @@ public class HFileInfo {
private static final String RESERVED_PREFIX = "hfile.";
static final UTF8StringKey LAST_KEY =
new UTF8StringKey(RESERVED_PREFIX + "LASTKEY");
- private static final UTF8StringKey FILE_CREATION_TIME_TS =
+ static final UTF8StringKey FILE_CREATION_TIME_TS =
new UTF8StringKey(RESERVED_PREFIX + "CREATE_TIME_TS");
- private static final UTF8StringKey KEY_VALUE_VERSION =
+ static final UTF8StringKey KEY_VALUE_VERSION =
new UTF8StringKey("KEY_VALUE_VERSION");
static final UTF8StringKey MAX_MVCC_TS_KEY =
new UTF8StringKey("MAX_MEMSTORE_TS_KEY");
-
- private static final int KEY_VALUE_VERSION_WITH_MVCC_TS = 1;
+ static final UTF8StringKey AVG_KEY_LEN =
+ new UTF8StringKey(RESERVED_PREFIX + "AVG_KEY_LEN");
+ static final UTF8StringKey AVG_VALUE_LEN =
+ new UTF8StringKey(RESERVED_PREFIX + "AVG_VALUE_LEN");
+ static final int KEY_VALUE_VERSION_WITH_MVCC_TS = 1;
private final Map<UTF8StringKey, byte[]> infoMap;
private final long fileCreationTime;
private final Option<Key> lastKey;
+ // This is set to trigger the constant MVCC 0,
+ // such that table version 6 and >=8 can have
+ // the same behavior.
private final long maxMvccTs;
- private final boolean containsMvccTs;
public HFileInfo(Map<UTF8StringKey, byte[]> infoMap) {
this.infoMap = infoMap;
this.fileCreationTime = parseFileCreationTime();
this.lastKey = parseLastKey();
this.maxMvccTs = parseMaxMvccTs();
- this.containsMvccTs = maxMvccTs > 0;
- if (containsMvccTs) {
- // The HFile written by Hudi does not contain MVCC timestamps.
- // Parsing MVCC timestamps is not supported.
- throw new UnsupportedOperationException("HFiles with MVCC timestamps are
not supported");
- }
Review Comment:
Sounds good. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
