Jackie-Jiang commented on a change in pull request #7629:
URL: https://github.com/apache/pinot/pull/7629#discussion_r735849755
##########
File path:
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/MultiValueVarByteRawIndexCreator.java
##########
@@ -67,13 +67,15 @@ public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType
* @param totalDocs Total number of documents to index
* @param valueType Type of the values
* @param maxRowLengthInBytes the size in bytes of the largest row, the
chunk size cannot be smaller than this
+ * @param maxNumberOfElements the maximum number of elements in a row
* @param writerVersion writer format version
*/
- public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType compressionType,
- String column, int totalDocs, DataType valueType, int writerVersion, int
maxRowLengthInBytes)
+ public MultiValueVarByteRawIndexCreator(File baseIndexDir,
ChunkCompressionType compressionType, String column,
+ int totalDocs, DataType valueType, int writerVersion, int
maxRowLengthInBytes, int maxNumberOfElements)
throws IOException {
//we will prepend the actual content with numElements and length array
containing length of each element
- int totalMaxLength = Integer.BYTES + Math.max(maxRowLengthInBytes,
TARGET_MAX_CHUNK_SIZE);
+ int maxLengthPrefixes = Integer.BYTES * maxNumberOfElements;
+ int totalMaxLength = Integer.BYTES + Math.max(maxRowLengthInBytes +
maxLengthPrefixes, TARGET_MAX_CHUNK_SIZE);
Review comment:
I still think this should be `Integer.BYTES + maxLengthPrefixes +
maxRowLengthInBytes` which is the maximum possible length of one single entry.
We don't want to have only one entry per chunk if the entry is small
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
