Venki Korukanti created DRILL-2178:
--------------------------------------
Summary: Change the outgoing record batch size in PartitionSender
Key: DRILL-2178
URL: https://issues.apache.org/jira/browse/DRILL-2178
Project: Apache Drill
Issue Type: Bug
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Currently we allocate memory for vectors in partition sender
OutgoingRecordBatch using allocateNew() which for most ValueVectors allocates
space for 4096 record capacity, but we flush the current record batch as soon
as we reach 1000 records causing wasted memory. Automatic resizing kicks in
after flushing few record batches, but auto resize always doubles or halves the
capacity. This cause the buffer record capacity to flip between 512 and 2048.
This JIRA is to:
1. Decide on the outgoing record batch depending upon the number of receivers
of partition sender. Default value is 1024, but when the number of receivers
exceeds 1000 change it to 512.
2. Allocate value vector space for storing the outgoing record batch size
decided in (1). For this we make use of {{AllocationHelper.allocate(ValueVector
v, int valueCount, int bytesPerValue, int repeatedPerTop)}}. {{bytesPerValue}}
and {{repeatedPerTop}} is currently hard coded to 50 and 10, but this shouldn't
matter as these values are applicable for variable and repeated vectors which
have realloc facility if they run out of space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
