-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51578/
-----------------------------------------------------------
(Updated Sept. 7, 2016, 4:57 p.m.)
Review request for Ambari, Di Li, DIPAYAN BHOWMICK, and Pallav Kulshreshtha.
Changes
-------
There is some issue with "update diff", as one of the files in the patch
"ConfigurationBuilder" shows not patched cleanly.
This same patch was tested by Hadoop QA (+1 result) on the Jira.
For reference, I have also attached the same patch here as
"AMBARI-18071-Sep6.patch".
Bugs: AMBARI-18071
https://issues.apache.org/jira/browse/AMBARI-18071
Repository: ambari
Description
-------
When HDFS is configured with Encryption Zones, Files View to browse files will
give "No KeyProvider" error.
Steps to reproduce this issue:
1. Configure an encrypted zone in HDFS (Transparent Data Encryption). I have
used Hadoop's KMS (installed tar manually).
2. Create a Files View instance and provide a user/group with the privilege to
use the instance.
3. Log into the Ambari console as the user with the Files View permission.
4. Open the Files View instance.
5. Go to the folder which is configured as an encrypted zone.
6. Try to open an existing file in this folder.
7. This throws an error - java.io.IOException: No KeyProvider is configured,
cannot access an encrypted file.
8. When trying through the shell, opening this file works.
This happens because Files View doesn't have enough configuration set to browse
secured zone. Files view doesn't even provide an option to add this
configuration. This is why we see errors "No KeyProvider is configured, cannot
access an encrypted file".
To work around this, you could download client configuration from HDFS service
tab, and copy the core-site.xml and hdfs-site.xml files to
/etc/ambari-server/conf, then restart ambari-server. After this, the user is
able to open the file in the encrypted zone. Basically, the property
"dfs.encryption.key.provider.uri" which provides details of the KeyProvider, is
obtained from the hdfs-site.xml present in the /etc/ambari-server/conf folder.
Diffs
-----
contrib/views/utils/src/main/java/org/apache/ambari/view/utils/hdfs/ConfigurationBuilder.java
d505e9a
contrib/views/utils/src/test/java/org/apache/ambari/view/utils/hdfs/ConfigurationBuilderTest.java
ed3025b
Diff: https://reviews.apache.org/r/51578/diff/
Testing
-------
**CAUSE:**
==========
The error of "No KeyProvider is configured" is seen only for those cases when
the HDFS uses DistributedFileSystem for its communication. When HDFS uses
WebHDFSFileSystem for communication, this error is not seen and the Ambari View
instance is able to open the files in the encrypted zones.
Why Ambari Views use either Distributed or WebHDFS file systems is explained
below:
Ambari views can be created using one of the 3 modes of configuration:
1. Local cluster
2. Remote cluster
3. Custom configuration (no cluster is associated here).
The HDFS works through abstraction. For Ambari Views, the actual file system
used during execution depends on whether the view instance was created using a
Local/Remote cluster or using Custom configuration. For instances created using
Local/Remote cluster, HDFS uses Distributed File System and for instances
created using Custom configuration, HDFS uses WebHDFSFileSystem.
WebHDFSFileSystem is an integrated part of the HDFS ecosystem. It is aware of
all the HDFS configuration. For this reason, when a KMS is configured in HDFS,
WebHDFSFileSystem is aware of the KeyProvider and no special config mapping is
needed. Thus, even the view instance created using Custom configuration doesn't
need any special configuration and can talk to the Encryption Zones
successfully.
However, for view instances created using Local/Remote cluster configuration,
HDFS uses the Distributed FileSystem. This Distributed FileSystem works as an
HDFS client and hence, is not fully aware of all the HDFS configuration. We
need to explicitly provide HDFS properties like
"dfs.encryption.key.provider.uri" to these ambari view instances to provide
details of the KeyProvider. The proposed fix helps in providing this property
value to the view as follows.
**FIX:**
========
The proposed fix (attached as "AMBARI-18071.patch") checks if the current view
instance configuration has any cluster associated in its context. If there is
an associated cluster then the instance has a Local/Remote cluster
configuration and needs to be provided with the HDFS KeyProvider information.
Otherwise, the WebHDFSFileSystem will take care of the KeyProvider if KMS is
configured.
To provide the property information, the parseProperties() in
ConfigurationBuilder.java looked best as we also set the defaultFS property
here. If a cluster is associated with the context, and if the property
"dfs.encryption.key.provider.uri" is not null, then this property is set in the
Configuration object and thus made available to Distributed file system of HDFS.
The Ambari VIew instance works successfully with both Local and Remote
configurations.
One more point to note in the configuration aspect is the addition of proxyuser
to the kms-site.xml for the ambari-server daemon. Without this proxyuser even
the custom configuration will not work. (I had installed hadoop's KMS on the
ambari-server manually)
**TESTING:**
============
I have done manual testing.
Log in as a user who is allowed to access encrypted zones.
Go to the Ambari View instance (with Local/Remote cluster configuration).
Open the encrypted zone folder and open an existing file in this directory.
The user can successfully preview the file without the "No KeyProvider
configured" error.
File Attachments (updated)
----------------
AMBARI-18071-Sep6.patch
https://reviews.apache.org/media/uploaded/files/2016/09/07/ca2655a2-7ad1-4af8-abee-b952b4040e4b__AMBARI-18071-Sep6.patch
Thanks,
Keta Patel
