[jira] [Commented] (HIVE-17327) LLAP IO: restrict native file ID usage to default FS to avoid hypothetical collisions when HDFS federation is used

[Peter Vary (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138204#comment-16138204
 ] 

Peter Vary commented on HIVE-17327:
-----------------------------------

[~gopalv],  [~sershe]: I think this patch breaks these tests:
{code}
org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistCpAs (batchId=250)
org.apache.hadoop.hive.common.TestFileUtils.testCopyWithDistcp (batchId=250)
{code}

Shall we remove the optional import here:
{code}
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>${hadoop.version}</version>
      <optional>true</optional>                    <-- This causes the problem
         <exclusions>
              <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
          </exclusion>
          <exclusion>
            <groupId>commmons-logging</groupId>
            <artifactId>commons-logging</artifactId>
          </exclusion>
         </exclusions>
    </dependency>
{code}

Also, I know it is used elsewhere too, but {{DistributedFileSystem}} is not a 
public api, especially not the {{DistributedFileSystem.getClient()}} which is 
marked private, and visible for testing. See: 
https://github.com/apache/hadoop/blob/branch-2.8.0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L1197

Shouldn't we raise a public API request from the hadoop community?

Thanks,
Peter

> LLAP IO: restrict native file ID usage to default FS to avoid hypothetical 
> collisions when HDFS federation is used
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-17327
>                 URL: https://issues.apache.org/jira/browse/HIVE-17327
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gopal V
>            Assignee: Sergey Shelukhin
>             Fix For: 3.0.0, 2.4.0
>
>         Attachments: HIVE-17327.01.patch, HIVE-17327.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)