Sankar Hariappan updated HIVE-17289:
    Attachment: HIVE-17289.01.patch

Added 01.patch with below changes.
- Used CopyUtils to copy files from ReplCopyTask (IMPORT/REPL LOAD)
- Set distcp doAs input as null in case of EXPORT and IMPORT flow. Will use the 
user config hive.distcp.privileged.doAs in case of REPL LOAD.
- Assumed lazy copy is set only for REPL LOAD and hence set doAs user input to 
hive.distcp.privileged.doAs if lazy copy is true and null if false. This is 
just to avoid passing this argument from multiple flows and also, the 
incremental REPL LOAD shares common code with IMPORT.
- Removed redundant code in ReplCopyTask/ReplCopyWork as it re-uses the 
CopyUtils implementation which does the same.
- Refactored ReplCopyTask.execute to properly distinguish code path for _files 
read and actual data files.
- Set the default value of hive.distcp.privileged.doAs to "hive".
- No tests added as the existing tests itself will cover the changes except 
distcp flow (due to hive.in.test) which needs to be tested manually.

> EXPORT and IMPORT shouldn't perform distcp with doAs privileged user.
> ---------------------------------------------------------------------
>                 Key: HIVE-17289
>                 URL: https://issues.apache.org/jira/browse/HIVE-17289
>             Project: Hive
>          Issue Type: Sub-task
>          Components: HiveServer2, repl
>    Affects Versions: 3.0.0
>            Reporter: Sankar Hariappan
>            Assignee: Sankar Hariappan
>              Labels: DR, Import, replication
>             Fix For: 3.0.0
>         Attachments: HIVE-17289.01.patch
> Currently, EXPORT uses distcp to dump data files to dump directory and IMPORT 
> uses distcp to copy the larger files/large number of files from dump 
> directory to table staging directory. But, this copy fails as distcp is 
> always done with doAs user specified in hive.distcp.privileged.doAs, which is 
> "hdfs' by default.
> Need to remove usage of doAs user when try to distcp from EXPORT/IMPORT flow.
> Privileged user based distcp should be done only for REPL DUMP/LOAD commands.
> Also, need to set the default config for hive.distcp.privileged.doAs to 
> "hive" as "hdfs" super-user is never allowed.

This message was sent by Atlassian JIRA

Reply via email to