[
https://issues.apache.org/jira/browse/DRILL-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447475#comment-16447475
]
ASF GitHub Bot commented on DRILL-6307:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1228#discussion_r183265234
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/record/TestRecordBatchSizer.java
---
@@ -45,7 +45,7 @@
private void verifyColumnValues(ColumnSize column, int
stdDataSizePerEntry, int stdNetSizePerEntry,
int dataSizePerEntry, int
netSizePerEntry, int totalDataSize,
int totalNetSize, int valueCount, int
elementCount,
- int estElementCountPerArray, boolean
isVariableWidth) {
+ int cardinality, boolean
isVariableWidth) {
--- End diff --
Maybe explain the cardinality here since I've seen it (or have used it) in
three different ways:
* Type cardinality: (Required: 1, Optional: (0, 1), Repeated: *)
* Batch cardinality: the number of rows in a batch. (Or, equivalently, the
number of top-level values in a vector.)
* Array cardinality: the number of values in an array (a single top-level
value which is repeated)
"Cardinality" is a good word, may just need disambiguation.
> Handle empty batches in record batch sizer correctly
> ----------------------------------------------------
>
> Key: DRILL-6307
> URL: https://issues.apache.org/jira/browse/DRILL-6307
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.13.0
> Reporter: Padma Penumarthy
> Assignee: Padma Penumarthy
> Priority: Major
> Fix For: 1.14.0
>
>
> when we get empty batch, record batch sizer calculates row width as zero. In
> that case, we do not do accounting and memory allocation correctly for
> outgoing batches.
> For example, in merge join, for outer left join, if right side batch is
> empty, we still have to include the right side columns as null in outgoing
> batch.
> Say first batch is empty. Then, for outgoing, we allocate empty vectors with
> zero capacity. When we read the next batch with data, we will end up going
> through realloc loop. If we use right side row width as 0 in outgoing row
> width calculation, number of rows we will calculate will be higher and later
> when we get a non empty batch, we might exceed the memory limits.
> One possible workaround/solution : Allocate memory based on std size for
> empty input batch. Use allocation width as width of the batch in number of
> rows calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
