[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580366#comment-16580366
]
Misha Dmitriev commented on HIVE-17684:
---------------------------------------
I've resumed working on this, and finally found the reason for the mysterious
failure described in my previous comment. It turns out that the following test
outputs:
./clientpositive/auto_join25.q.out
./clientpositive/auto_join_without_localtask.q.out
./clientpositive/infer_bucket_sort_convert_join.q.out
./clientpositive/mapjoin_hook.q.out
actually _do_ contain the "Hive Runtime Error: Map local work exhausted memory"
string, i.e. they expect the old memory exhaustion check to be triggered. With
my new code, this check doesn't get triggered anymore (probably correctly), so
these tests fail. I think a sufficient level of Hive expertise is needed to
determine how the expected output should look like in the new situation, so I
would prefer to leave it to you [~stakiar].
But looks like we still have a high number of failing CLI tests, that likely
fail because the "in tests" flag is not set when they run, and thus my new high
GC time get triggered. For these tests to pass, I guess we either need to make
all CLI tests set the "in tests" flag somehow, or find out another JVM property
or something that is set only when these tests run. Any suggestions?
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -----------------------------------------------------
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Misha Dmitriev
> Priority: Major
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
