[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048298#comment-17048298
]
Hive QA commented on HIVE-21971:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995015/HIVE-21971.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18095 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets]
(batchId=1)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20889/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20889/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20889/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995015 - PreCommit-HIVE-Build
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> -------------------------------------------------------------------------------------------------------
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 2.3.4
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
