[
https://issues.apache.org/jira/browse/HIVE-24737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278959#comment-17278959
]
Panagiotis Garefalakis edited comment on HIVE-24737 at 2/4/21, 4:44 PM:
------------------------------------------------------------------------
Hey David, seems like the obtainBufferSize value is afterwards used as
[TEZ_RUNTIME_UNORDERED_OUTPUT_BUFFER_SIZE_MB|https://github.com/apache/tez/blob/bf4942b49a3d9c0d326f64715b3e76952b77382d/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java#L203]
property.
I believe the conf is there in case the groupByOperator stats are completely
off -- creating an unreasonably large output buffer. Maybe we could remove it
and have a large upper limit instead, not sure what that value could be though
(default is 100MB) -- any ideas here [~abstractdog] ?
was (Author: pgaref):
Hey David, seems like the obtainBufferSize value is afterwards used as
[TEZ_RUNTIME_UNORDERED_OUTPUT_BUFFER_SIZE_MB|https://github.com/apache/tez/blob/bf4942b49a3d9c0d326f64715b3e76952b77382d/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/api/TezRuntimeConfiguration.java#L203]
property.
I believe the conf is there in case the groupByOperator stats are completely
off -- creating an unreasonably large output buffer. Maybe we could remove and
have a large upper limit instead, not sure what that value could be though
(default is 100MB) -- any ideas here [~abstractdog] ?
> Remove Configuration TEZ_SIMPLE_CUSTOM_EDGE_TINY_BUFFER_SIZE_MB
> ---------------------------------------------------------------
>
> Key: HIVE-24737
> URL: https://issues.apache.org/jira/browse/HIVE-24737
> Project: Hive
> Issue Type: Improvement
> Reporter: David Mollitor
> Priority: Minor
>
> Please remove {{TEZ_SIMPLE_CUSTOM_EDGE_TINY_BUFFER_SIZE_MB}}
>
> It is never in practice actually used. Can it just be assigned a sensible
> hard-coded value?
>
> This seem like an over optimization at the cost of yet another configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
