-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/
-----------------------------------------------------------
(Updated April 14, 2016, 6:39 p.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay
Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan
Periasamy.
Changes
-------
Addressed review comments and tested setup script for all db flavor in local
environment
Bugs: RANGER-900
https://issues.apache.org/jira/browse/RANGER-900
Repository: ranger
Description
-------
**Problem Statement :**
Remove option to store audit to DB as storing audit logs in db requires lots of
data management activity and frequent backup-restore process might hamper
Ranger application performance. Production team might face frequent down-time
issues due to db disk space reclaim activities.
**Proposed Solution :**
Proposed solution is having below mentioned approch :
1. Remove audit to DB related properties from install.properties of all
components.
2. Disable shell script code to read audit to DB related properties from
install.properties of all components.
3. Disable code from dba_script.py which is invoked to create audit DB schema,
audit User and executes grants privileges.
4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and
stop executing audit to Db related sql patches.
5. Make solr as mandatory audit data store/source.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java
3e89cc4
agents-common/scripts/enable-agent.sh b9511d2
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java
8ee6bea
hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
hbase-agent/scripts/install.properties 795ea3e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
hdfs-agent/scripts/install.properties b4dda13
hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
hive-agent/scripts/install.properties 6b71a85
kms/scripts/install.properties 7762948
knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
knox-agent/scripts/install.properties 1febd49
plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
plugin-kafka/scripts/install.properties 79ea6db
plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
plugin-solr/scripts/install.properties a3d9887
plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
plugin-yarn/scripts/install.properties 3780068
security-admin/scripts/db_setup.py 3d20fcd
security-admin/scripts/dba_script.py 0ebd90b
security-admin/scripts/install.properties 1d9d207
security-admin/scripts/setup.sh bf29ed6
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml
3333827
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
storm-agent/scripts/install.properties f2aa5c4
Diff: https://reviews.apache.org/r/45091/diff/
Testing
-------
**Steps Performaed (With patch) :**
**Use-case 1:**
Fresh Ranger Admin Installation :
Steps:
1. After Ranger installation did not find any audit to DB related properties in
install.properties file so provided 'solr' as audit data store and configured
solr URL in solr_url property.
2. Executed setup.sh to install Ranger
Expected Behaviour :
1. Installation script should complete successfully and after starting Ranger,
Ranger UI should work; user should able to create services, policies, users and
groups.
Actual Behaviour :
1. In installation log it was observed that installation process skipped
creation of audit DB, audit user and execution of audit db related sql patches.
2. Ranger installation was finished successfully.
3. After starting Ranger; was able to login to Ranger and Ranger UI was working
fine. Was able to create services, policies, users and groups.
**Use-case 2:**
Enabling Ranger plugin and writing audit logs To solr :
Steps:
1. Enabled HDFS plugin with solr and provided solr url so that hdfs component
should write audit logs in solr.
2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger
policies were created.
Expected Behaviour :
HDFS plugin should write audit logs to provided solr url and same logs should
appear in Ranger admin UI -> Audit menu -> Access tab.
Actual Behaviour :
Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
**Use-case 3:**
Ranger admin and Ranger plugins upgrade:
Steps:
1. Installed Ranger admin without patch and started Ranger admin with audit
source as DB; enabled HDFS plugin with Audit logs to all three audit
destination DB, HDFS and solr.
2. Created HDFS service and policies; assigned policies to users with different
combination of access permissions.
3. From console window excecuted HDFS command to READ/CREATE HDFS resources on
which Ranger policies was created.
4. It was observed that HDFS plugin was writing audit logs to all three audit
stores.
5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger
Admin, Stopped HDFS component.
7. Copied Ranger admin install.properties and installed Ranger with patch and
used same properties of previous installation, since new install.properties did
not have audit to DB related properties so skipped that and provided solr url
which was used in solr related config of HDFS plugin.
7. Executed Ranger setup script and restarted Ranger admin.
8. Now Ranger UI was reading audit logs from solr source and expected logs were
available in Ranger Admin UI -> Audit menu -> Access tab.
9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies
exist.
Expected Behaviour :
1. HDFS plugin should write new logs to provided solr url and same logs should
appear in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs should appear in Solr UI also.
3. HDFS plugin must not write any new audit logs to DB.
Actual Behaviour :
1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs were available in solr UI.
3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
Thanks,
Pradeep Agrawal
