[
https://issues.apache.org/jira/browse/SENTRY-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536756#comment-14536756
]
Binglin Chang commented on SENTRY-705:
--------------------------------------
Add some logging and found the root cause. HiveAuthzBinding.set is called twice
when compile(each with different queryId), but in execute, clear can only
remove the last added object in authzBindingMap. so each successful query will
end up add a HiveAuthzBinding object to authzBindingMap, never got freed.
{noformat}
2015-05-10 00:13:49,414 WARN [HiveServer2-Handler-Pool: Thread-28]:
authz.HiveAuthzBinding (HiveAuthzBinding.java:set(157)) - authzBindingMap.set
3acea9e4-a715-4890-b051-b473f242a1ff_3
java.lang.Exception
at
org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at
org.apache.sentry.binding.hive.HiveAuthzBindingHook.authorizeWithHiveBindings(HiveAuthzBindingHook.java:527)
at
org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:326)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,415 WARN [HiveServer2-Handler-Pool: Thread-28]:
authz.HiveAuthzBinding (HiveAuthzBinding.java:set(157)) - authzBindingMap.set
3acea9e4-a715-4890-b051-b473f242a1ff_4
java.lang.Exception
at
org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at
org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:343)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,424 INFO [HiveServer2-Background-Pool: Thread-50]:
ql.Driver (Driver.java:execute(1206)) - Starting command: select 1+1
2015-05-10 00:13:49,424 WARN [HiveServer2-Background-Pool: Thread-50]:
authz.HiveAuthzBinding (HiveAuthzBinding.java:clear(169)) -
authzBindingMap.remove 3acea9e4-a715-4890-b051-b473f242a1ff_4
{noformat}
> Possible memory leak of HiveAuthzBinding.authzBindingMap
> --------------------------------------------------------
>
> Key: SENTRY-705
> URL: https://issues.apache.org/jira/browse/SENTRY-705
> Project: Sentry
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Binglin Chang
>
> We are using hive0.13 with sentry, version cdh5.2.0, noticed that hiveserver
> uses lot of memory and doesn't get freed.
> Memory profiling showed that there are lots(millions) of HiveConf and
> HiveAuthzConf instances hold by HiveAuthzBinding.authzBindingMap
> Look at the code, related call order:
> Driver.compile
> HiveAuthzBindingHook.preAnalyze
> HiveAuthzBindingHook.postAnalyze
> hiveAuthzBinding.set
> Driver.execute
> HiveAuthzBindingPreExecHook.run
> hiveAuthzBinding.clear
> Between Driver.compile and Driver.execute, there are too many logic(which
> sentry can not control), not sure hiveAuthzBinding.clear is guaranteed to be
> called
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
