[jira] [Commented] (HIVE-10233) Hive on tez: memory manager for grace hash join

[Mostafa Mokhtar (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597139#comment-14597139
 ] 

Mostafa Mokhtar commented on HIVE-10233:
----------------------------------------

[~gunther]
Should totalAvailableMemory be HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD?
resourceAvailable.getMemory() would basically return container size, this will 
result in a lot of GC if all memory gets used up. 

Change Log.Debug to Log.INFO and print the input size. 
{code}
 for (MapJoinOperator mj : mapJoins) {
                mj.getConf().setMemoryNeeded(minMemory);
                  LOG.info("Setting " + minMemory + " bytes needed for " + mj);
              }
{code}

Also I am not following the logic here, shouldn't the memory needed per 
operator be something like "(estimate size) / (Total input sizes) x 
memoryAvailable" ?
{code}
              int numJoins = mapJoins.size();
              long minMemory = totalAvailableMemory / ((numJoins > 0) ? 
numJoins : 1);
              minMemory = Math.min(minMemory, onePercentMemory);
        
              for (MapJoinOperator mj : mapJoins) {
                mj.getConf().setMemoryNeeded(minMemory);
                if (LOG.isDebugEnabled()) {
                  LOG.debug("Setting " + minMemory + " bytes needed for " + mj);
                }
              }
{code}

> Hive on tez: memory manager for grace hash join
> -----------------------------------------------
>
>                 Key: HIVE-10233
>                 URL: https://issues.apache.org/jira/browse/HIVE-10233
>             Project: Hive
>          Issue Type: Bug
>          Components: Tez
>    Affects Versions: llap, 2.0.0
>            Reporter: Vikram Dixit K
>            Assignee: Gunther Hagleitner
>         Attachments: HIVE-10233-WIP-2.patch, HIVE-10233-WIP-3.patch, 
> HIVE-10233-WIP-4.patch, HIVE-10233-WIP-5.patch, HIVE-10233-WIP-6.patch, 
> HIVE-10233-WIP-7.patch, HIVE-10233-WIP-8.patch, HIVE-10233.08.patch, 
> HIVE-10233.09.patch, HIVE-10233.10.patch, HIVE-10233.11.patch
>
>
> We need a memory manager in llap/tez to manage the usage of memory across 
> threads. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)