[
https://issues.apache.org/jira/browse/HIVE-25685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-25685:
--------------------------------
Description:
After some debugging I found that in the following codepath configureJobConf is
called to prepare the jobConf object for special storage handlers, however, if
the jobConf hasn't contained the needed hbase props, then this code path
doesn't make sure of adding that, even if the method name suggests.
{code:java}
HBaseConfiguration.addHbaseResources(Configuration) line: 82
HBaseConfiguration.create() line: 98
HBaseConfiguration.create(Configuration) line: 107
HBaseStorageHandler.setConf(Configuration) line: 134
ReflectionUtils.setConf(Object, Configuration) line: 77
ReflectionUtils.newInstance(Class<T>, Configuration) line: 137
HiveUtils.getStorageHandler(Configuration, String) line: 299
PlanUtils.configureJobConf(TableDesc, JobConf) line: 995
MapWork.configureJobConf(JobConf) line: 661
TezWork.configureJobConfAndExtractJars(JobConf) line: 346
TezTask.execute() line: 186
TezTask(Task<T>).executeTask(HiveHistory) line: 213
{code}
Here, addHBaseResources adds hbase xml configs to HBaseStorageHandler.hbaseConf
during the reflection flow, but then they're not added to the passed jobConf in
configureJobConf.
This can cause problems in secure clusters under some circumstances, where
hbase props are not present in the conf which is used in TezTask.execute.
was:
After some debugging I found that in the following codepath configureJobConf is
called to prepare the jobConf object for special storage handlers, however, if
the jobConf hasn't contained the needed hbase props, then this code path
doesn't make sure of adding that, even if the method name suggests.
{code}
HBaseConfiguration.addHbaseResources(Configuration) line: 82
HBaseConfiguration.create() line: 98
HBaseConfiguration.create(Configuration) line: 107
HBaseStorageHandler.setConf(Configuration) line: 134
ReflectionUtils.setConf(Object, Configuration) line: 77
ReflectionUtils.newInstance(Class<T>, Configuration) line: 137
HiveUtils.getStorageHandler(Configuration, String) line: 299
PlanUtils.configureJobConf(TableDesc, JobConf) line: 995
MapWork.configureJobConf(JobConf) line: 661
TezWork.configureJobConfAndExtractJars(JobConf) line: 346
TezTask.execute() line: 186
TezTask(Task<T>).executeTask(HiveHistory) line: 213
{code}
This can cause problems in secure clusters under some circumstances, where
hbase props are not present in the conf which is used in TezTask.execute.
> HBaseStorageHandler: ensure that hbase properties are present in final
> JobConf for Tez
> --------------------------------------------------------------------------------------
>
> Key: HIVE-25685
> URL: https://issues.apache.org/jira/browse/HIVE-25685
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> After some debugging I found that in the following codepath configureJobConf
> is called to prepare the jobConf object for special storage handlers,
> however, if the jobConf hasn't contained the needed hbase props, then this
> code path doesn't make sure of adding that, even if the method name suggests.
>
> {code:java}
> HBaseConfiguration.addHbaseResources(Configuration) line: 82
> HBaseConfiguration.create() line: 98
> HBaseConfiguration.create(Configuration) line: 107
> HBaseStorageHandler.setConf(Configuration) line: 134
> ReflectionUtils.setConf(Object, Configuration) line: 77
> ReflectionUtils.newInstance(Class<T>, Configuration) line: 137
> HiveUtils.getStorageHandler(Configuration, String) line: 299
> PlanUtils.configureJobConf(TableDesc, JobConf) line: 995
> MapWork.configureJobConf(JobConf) line: 661
> TezWork.configureJobConfAndExtractJars(JobConf) line: 346
> TezTask.execute() line: 186
> TezTask(Task<T>).executeTask(HiveHistory) line: 213
> {code}
> Here, addHBaseResources adds hbase xml configs to
> HBaseStorageHandler.hbaseConf during the reflection flow, but then they're
> not added to the passed jobConf in configureJobConf.
> This can cause problems in secure clusters under some circumstances, where
> hbase props are not present in the conf which is used in TezTask.execute.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
