[
https://issues.apache.org/jira/browse/HIVE-25401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25401:
----------------------------------
Labels: pull-request-available (was: )
> Insert overwrite a table which location is on other cluster fail in
> kerberos cluster
> --------------------------------------------------------------------------------------
>
> Key: HIVE-25401
> URL: https://issues.apache.org/jira/browse/HIVE-25401
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.3.0, 3.1.2
> Environment: hive 2.3
> hadoop3 cluster with kerberos
> Reporter: Max Xie
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2021-07-29-14-25-23-418.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> we have tow hdfs clusters with kerberos security, it means that mapreduce
> task need delegation tokens to authenticate namenode when hive on mapreduce
> run.
> Insert overwrite a table which location is on other cluster fail in kerberos
> cluster. For example,
> # yarn cluster's default fs is hdfs://cluster1
> # tb1's location is hdfs://cluster1/tb1
> # tb2's location is hdfs://cluster2/tb2
> # sql `INSERT OVERWRITE TABLE tb2 SELECT * from tb1` run on yarn cluster
> will fail
>
> reduce task error log:
> !image-2021-07-29-14-25-23-418.png!
> How to fix:
> After dig it, web found mapreduce job just obtain delegation tokens for input
> files in FileInputFormat. But Hive context get extendal scratchDir base on
> table's location, If the table 's location is on other cluster, the
> delegation token will not be obtained.
> So we need to obtaine delegation tokens for hive scratchDirs before hive
> submit mapreduce job.
>
> How to test:
> no test
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
