This is an automatically generated e-mail. To reply, visit:

Ship it!

Ship It!

- Di Li

On Sept. 7, 2016, 4:57 p.m., Keta Patel wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51578/
> -----------------------------------------------------------
> (Updated Sept. 7, 2016, 4:57 p.m.)
> Review request for Ambari, Di Li, DIPAYAN BHOWMICK, and Pallav Kulshreshtha.
> Bugs: AMBARI-18071
>     https://issues.apache.org/jira/browse/AMBARI-18071
> Repository: ambari
> Description
> -------
> When HDFS is configured with Encryption Zones, Files View to browse files 
> will give "No KeyProvider" error.
> Steps to reproduce this issue:
> 1. Configure an encrypted zone in HDFS (Transparent Data Encryption). I have 
> used Hadoop's KMS (installed tar manually).
> 2. Create a Files View instance and provide a user/group with the privilege 
> to use the instance.
> 3. Log into the Ambari console as the user with the Files View permission.
> 4. Open the Files View instance.
> 5. Go to the folder which is configured as an encrypted zone.
> 6. Try to open an existing file in this folder.
> 7. This throws an error - java.io.IOException: No KeyProvider is configured, 
> cannot access an encrypted file. 
> 8. When trying through the shell, opening this file works.
> This happens because Files View doesn't have enough configuration set to 
> browse secured zone. Files view doesn't even provide an option to add this 
> configuration. This is why we see errors "No KeyProvider is configured, 
> cannot access an encrypted file". 
> To work around this, you could download client configuration from HDFS 
> service tab, and copy the core-site.xml and hdfs-site.xml files to 
> /etc/ambari-server/conf, then restart ambari-server. After this, the user is 
> able to open the file in the encrypted zone. Basically, the property 
> "dfs.encryption.key.provider.uri" which provides details of the KeyProvider, 
> is obtained from the hdfs-site.xml present in the /etc/ambari-server/conf 
> folder.
> Diffs
> -----
> contrib/views/utils/src/main/java/org/apache/ambari/view/utils/hdfs/ConfigurationBuilder.java
>  d505e9a 
> contrib/views/utils/src/test/java/org/apache/ambari/view/utils/hdfs/ConfigurationBuilderTest.java
>  ed3025b 
> Diff: https://reviews.apache.org/r/51578/diff/
> Testing
> -------
> **CAUSE:**
> ==========
> The error of "No KeyProvider is configured" is seen only for those cases when 
> the HDFS uses DistributedFileSystem for its communication. When HDFS uses 
> WebHDFSFileSystem for communication, this error is not seen and the Ambari 
> View instance is able to open the files in the encrypted zones.
> Why Ambari Views use either Distributed or WebHDFS file systems is explained 
> below:
> Ambari views can be created using one of the 3 modes of configuration:
> 1. Local cluster
> 2. Remote cluster
> 3. Custom configuration (no cluster is associated here).
> The HDFS works through abstraction. For Ambari Views, the actual file system 
> used during execution depends on whether the view instance was created using 
> a Local/Remote cluster or using Custom configuration. For instances created 
> using Local/Remote cluster, HDFS uses Distributed File System and for 
> instances created using Custom configuration, HDFS uses WebHDFSFileSystem.
> WebHDFSFileSystem is an integrated part of the HDFS ecosystem. It is aware of 
> all the HDFS configuration. For this reason, when a KMS is configured in 
> HDFS, WebHDFSFileSystem is aware of the KeyProvider and no special config 
> mapping is needed. Thus, even the view instance created using Custom 
> configuration doesn't need any special configuration and can talk to the 
> Encryption Zones successfully.
> However, for view instances created using Local/Remote cluster configuration, 
> HDFS uses the Distributed FileSystem. This Distributed FileSystem works as an 
> HDFS client and hence, is not fully aware of all the HDFS configuration. We 
> need to explicitly provide HDFS properties like 
> "dfs.encryption.key.provider.uri" to these ambari view instances to provide 
> details of the KeyProvider. The proposed fix helps in providing this property 
> value to the view as follows.
> **FIX:**
> ========
> The proposed fix (attached as "AMBARI-18071.patch") checks if the current 
> view instance configuration has any cluster associated in its context. If 
> there is an associated cluster then the instance has a Local/Remote cluster 
> configuration and needs to be provided with the HDFS KeyProvider information. 
> Otherwise, the WebHDFSFileSystem will take care of the KeyProvider if KMS is 
> configured.
> To provide the property information, the parseProperties() in 
> ConfigurationBuilder.java looked best as we also set the defaultFS property 
> here. If a cluster is associated with the context, and if the property 
> "dfs.encryption.key.provider.uri" is not null, then this property is set in 
> the Configuration object and thus made available to Distributed file system 
> of HDFS.
> The Ambari VIew instance works successfully with both Local and Remote 
> configurations.
> One more point to note in the configuration aspect is the addition of 
> proxyuser to the kms-site.xml for the ambari-server daemon. Without this 
> proxyuser even the custom configuration will not work. (I had installed 
> hadoop's KMS on the ambari-server manually)
> **TESTING:**
> ============
> I have done manual testing.
> Log in as a user who is allowed to access encrypted zones.
> Go to the Ambari View instance (with Local/Remote cluster configuration).
> Open the encrypted zone folder and open an existing file in this directory.
> The user can successfully preview the file without the "No KeyProvider 
> configured" error.
> File Attachments
> ----------------
> AMBARI-18071-Sep6.patch
> https://reviews.apache.org/media/uploaded/files/2016/09/07/ca2655a2-7ad1-4af8-abee-b952b4040e4b__AMBARI-18071-Sep6.patch
> Thanks,
> Keta Patel

Reply via email to