[
https://issues.apache.org/jira/browse/HIVE-21329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778975#comment-16778975
]
Gopal V commented on HIVE-21329:
--------------------------------
LGTM - +1
{code}
2019-02-27 01:35:38,588 [INFO] [TezChild]
|writers.UnorderedPartitionedKVWriter|: Reducer_2: numBuffers=2,
sizePerBuffer=524288, skipBuffers=true, numPartitions=1,
availableMemory=1048576, maxSingleBufferSizeBytes=2147483647,
pipelinedShuffle=false, isFinalMergeEnabled=false, numPartitions=1,
reportPartitionStats=ENABLED
2019-02-27 01:35:38,592 [INFO] [TezChild] |tez.MapRecordProcessor|: Main input
name is Map 1
{code}
The allocation does show up in the timings in quick tests.
{code}
0: jdbc:hive2://localhost:10013/> set
hive.tez.unordered.output.buffer.size.mb=1024;
0: jdbc:hive2://localhost:10013/> select count(1) from store_sales order by 1;
INFO : Completed executing
command(queryId=gopal_20190227015023_75eeb0ec-edd2-4248-8122-a3147c0bb502);
Time taken: 5.436 seconds
{code}
with -1
{code}
INFO : Completed executing
command(queryId=gopal_20190227015049_2068d1e9-b9ac-463a-8bdb-738966a480a7);
Time taken: 3.645 seconds
{code}
> Custom Tez runtime unordered output buffer size depending on operator pipeline
> ------------------------------------------------------------------------------
>
> Key: HIVE-21329
> URL: https://issues.apache.org/jira/browse/HIVE-21329
> Project: Hive
> Issue Type: Improvement
> Components: Tez
> Reporter: Jesus Camacho Rodriguez
> Assignee: Jesus Camacho Rodriguez
> Priority: Major
> Attachments: HIVE-21329.patch
>
>
> For instance, if we have a reduce sink operator with no keys followed by a
> Group By (merge partial), we can decrease the output buffer size since we
> will only produce a single row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
