richardstartin commented on a change in pull request #7629:
URL: https://github.com/apache/pinot/pull/7629#discussion_r735963251
##########
File path:
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/MultiValueVarByteRawIndexCreator.java
##########
@@ -67,26 +67,23 @@ public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType
* @param totalDocs Total number of documents to index
* @param valueType Type of the values
* @param maxRowLengthInBytes the size in bytes of the largest row, the
chunk size cannot be smaller than this
+ * @param maxNumberOfElements the maximum number of elements in a row
* @param writerVersion writer format version
*/
- public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType compressionType,
- String column, int totalDocs, DataType valueType, int writerVersion, int
maxRowLengthInBytes)
+ public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType compressionType, String column,
+ int totalDocs, DataType valueType, int writerVersion, int
maxRowLengthInBytes, int maxNumberOfElements)
throws IOException {
//we will prepend the actual content with numElements and length array
containing length of each element
- int totalMaxLength = Integer.BYTES + Math.max(maxRowLengthInBytes,
TARGET_MAX_CHUNK_SIZE);
+ int maxLengthPrefixes = Integer.BYTES * maxNumberOfElements;
+ int totalMaxLength = Integer.BYTES + maxRowLengthInBytes +
maxLengthPrefixes;
File file = new File(baseIndexDir,
column + Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION);
- int numDocsPerChunk = getNumDocsPerChunk(totalMaxLength);
- _indexWriter = new VarByteChunkSVForwardIndexWriter(file, compressionType,
totalDocs,
- numDocsPerChunk, totalMaxLength,
- writerVersion);
- _valueType = valueType;
- }
-
- private static int getNumDocsPerChunk(int lengthOfLongestEntry) {
int overheadPerEntry =
- lengthOfLongestEntry +
VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE;
- return Math.max(TARGET_MAX_CHUNK_SIZE / overheadPerEntry, 1);
+ maxRowLengthInBytes + maxLengthPrefixes +
VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE;
+ int numDocsPerChunk = Math.max(TARGET_MAX_CHUNK_SIZE / overheadPerEntry,
1);
Review comment:
Right, but I removed it because you said to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
