mayankshriv commented on a change in pull request #4348: A template
implementation for VarLengthBytesValueReaderWriter
URL: https://github.com/apache/incubator-pinot/pull/4348#discussion_r296040022
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/io/util/VarLengthBytesValueReaderWriter.java
##########
@@ -0,0 +1,87 @@
+package org.apache.pinot.core.io.util;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteOrder;
+import org.apache.pinot.core.io.writer.impl.OffHeapByteArrayStore;
+import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+
+
+public class VarLengthBytesValueReaderWriter implements Closeable, ValueReader
{
+
+ private OffHeapByteArrayStore _buffer;
+
+ // Used for memory-mapping while loading a segment
+ public VarLengthBytesValueReaderWriter(PinotDataBuffer pinotDataBuffer) {
+ PinotDataBuffer valuesBuffer = readHedader(pinotDataBuffer);
+ _buffer = new OffHeapByteArrayStore(valuesBuffer);
+ }
+
+ // Used to create one while building a segment
+ public VarLengthBytesValueReaderWriter(byte[][] arrays, File dictionaryFile,
long size) throws IOException {
+ long sizeWithHeader = size + getHeaderSize(arrays);
+ PinotDataBuffer pinotDataBuffer = PinotDataBuffer
+ .mapFile(dictionaryFile, false, 0, size, ByteOrder.BIG_ENDIAN,
+ getClass().getSimpleName());
+ writeHeader(pinotDataBuffer, arrays);
+ _buffer = new OffHeapByteArrayStore(pinotDataBuffer);
+ for (byte[] array : arrays) {
+ _buffer.add(array);
+ }
+ }
+
+ private int getHeaderSize(byte[][] arrays) {
+ return 11; // or whatever computed value for fixed/var sized headers,
version etc..
+ }
+
+ private void writeHeader(PinotDataBuffer pinotDataBuffer, byte[][] arrays) {
+ // write headers here, maybe header has some metadata about values.
+ }
+
+ private PinotDataBuffer readHedader(PinotDataBuffer pinotDataBuffer) {
Review comment:
Typo in name `s/readHedader/readHeader`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
