[
https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096055#comment-15096055
]
Hive QA commented on HIVE-12837:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781917/HIVE-12837.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 9965 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_joins_explain.q-vector_decimal_aggregate.q-vector_groupby_mapjoin.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6610/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6610/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12781917 - PreCommit-HIVE-TRUNK-Build
> Better memory estimation/allocation for hybrid grace hash join during hash
> table loading
> ----------------------------------------------------------------------------------------
>
> Key: HIVE-12837
> URL: https://issues.apache.org/jira/browse/HIVE-12837
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.1.0
> Reporter: Wei Zheng
> Assignee: Wei Zheng
> Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch
>
>
> This is to avoid an edge case when the memory available is very little (less
> than a single write buffer size), and we start loading the hash table. Since
> the write buffer is lazily allocated, we will easily run out of memory before
> even checking if we should spill any hash partition.
> e.g.
> Total memory available: 210 MB
> Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB
> Size of write buffer: 8 MB (lazy allocation)
> Number of hash partitions: 16
> Number of hash partitions created in memory: 13
> Number of hash partitions created on disk: 3
> Available memory left after HybridHashTableContainer initialization:
> 210-16*13=2MB
> Now let's say a row is to be loaded into a hash partition in memory, it will
> try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM.
> Solution is to perform the check for possible spilling earlier so we can
> spill partitions if memory is about to be full, to avoid OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
