siddharthteotia commented on code in PR #9485:
URL: https://github.com/apache/pinot/pull/9485#discussion_r988357249
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/TransferableBlockUtils.java:
##########
@@ -43,4 +50,41 @@ public static TransferableBlock
getErrorTransferableBlock(Map<Integer, String> e
public static boolean isEndOfStream(TransferableBlock transferableBlock) {
return transferableBlock.isEndOfStreamBlock();
}
+
+ /**
+ * Split a block into multiple block so that each block size is within
maxBlockSize.
+ * Currently, we only support split for row type dataBlock.
+ * For columnar data block, we return the original data block.
+ * Metadata data block split is not supported.
+ *
+ * When row size is greater than maxBlockSize, we pack each row as a
separate block.
+ */
+ public static List<TransferableBlock> splitBlock(TransferableBlock block,
BaseDataBlock.Type type, int maxBlockSize) {
+ List<TransferableBlock> blockChunks = new ArrayList<>();
+ if (type != BaseDataBlock.Type.ROW) {
+ return Collections.singletonList(block);
+ } else {
+ int rowSizeInBytes = ((RowDataBlock)
block.getDataBlock()).getRowSizeInBytes();
+ int numRowsPerChunk = maxBlockSize / rowSizeInBytes;
+ Preconditions.checkState(numRowsPerChunk > 0, "row size too large for
query engine to handle, abort!");
Review Comment:
Can we include the offending rowSize in the message / exception ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
