[ https://issues.apache.org/jira/browse/DRILL-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310082#comment-14310082 ]
Steven Phillips commented on DRILL-2178: ---------------------------------------- +1 > Update outgoing record batch size and allocation in PartitionSender > ------------------------------------------------------------------- > > Key: DRILL-2178 > URL: https://issues.apache.org/jira/browse/DRILL-2178 > Project: Apache Drill > Issue Type: Bug > Reporter: Venki Korukanti > Assignee: Venki Korukanti > Fix For: 0.8.0 > > Attachments: > 0001-DRILL-2178-Update-outgoing-record-batch-size-and-all.patch > > > Currently we allocate memory for vectors in partition sender > OutgoingRecordBatch using allocateNew() which for most ValueVectors allocates > space for 4096 record capacity, but we flush the current record batch as soon > as we reach 1000 records causing wasted memory. Automatic resizing kicks in > after flushing few record batches, but auto resize always doubles or halves > the capacity. This cause the buffer record capacity to flip between 512 and > 2048. > This JIRA is to: > 1. Decide on the outgoing record batch depending upon the number of receivers > of partition sender. Default value is 1024, but when the number of receivers > exceeds 1000 change it to 512. > 2. Allocate value vector space for storing the outgoing record batch size > decided in (1). For this we make use of > {{AllocationHelper.allocate(ValueVector v, int valueCount, int bytesPerValue, > int repeatedPerTop)}}. {{bytesPerValue}} and {{repeatedPerTop}} is currently > hard coded to 50 and 10, but this shouldn't matter as these values are > applicable for variable and repeated vectors which have realloc facility if > they run out of space. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)