IMPALA-3410 [DOCS] Rework Impala security topics to be generic This is part 1 of the changes being made to the Impala authorization topics. References to CDH and Cloudera Manager docs/products have been either 'hidden' or removed completely.
Examples with Sentry have been made more generic. Instances of Cloudera-specific folders or filenames have been removed. Change-Id: Ie5c4431f3236b18fc282343ed98513f0e578130e Reviewed-on: http://gerrit.cloudera.org:8080/5931 Reviewed-by: Jim Apple <[email protected]> Reviewed-by: John Russell <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/77208d36 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/77208d36 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/77208d36 Branch: refs/heads/master Commit: 77208d36520b1118ec66a674980d3e7b954a7b1c Parents: 4a7946d Author: Ambreen Kazi <[email protected]> Authored: Tue Feb 7 09:04:25 2017 -0800 Committer: Impala Public Jenkins <[email protected]> Committed: Thu Feb 9 22:19:44 2017 +0000 ---------------------------------------------------------------------- docs/shared/impala_common.xml | 7 +- docs/topics/impala_authorization.xml | 106 +++++++++++++-------------- docs/topics/impala_security.xml | 19 +++-- docs/topics/impala_security_install.xml | 3 +- docs/topics/impala_ssl.xml | 42 +++++------ 5 files changed, 84 insertions(+), 93 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/77208d36/docs/shared/impala_common.xml ---------------------------------------------------------------------- diff --git a/docs/shared/impala_common.xml b/docs/shared/impala_common.xml index 4309e84..1b8c171 100644 --- a/docs/shared/impala_common.xml +++ b/docs/shared/impala_common.xml @@ -605,16 +605,15 @@ under the License. Sentry logs all facts that lead up to authorization decisions at the debug level. If you do not understand why Sentry is denying access, the best way to debug is to temporarily turn on debug logging: <ul> - <li audience="Cloudera"> + <li audience="hidden"> In Cloudera Manager, add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph> to the logging settings for your service through the corresponding <uicontrol>Logging Safety Valve</uicontrol> field for the Impala, Hive Server 2, or Solr Server services. </li> <li> - On systems not managed by Cloudera Manager, add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph> - to the <filepath>log4j.properties</filepath> file on each host in the cluster, in the appropriate - configuration directory for each service. + Add <codeph>log4j.logger.org.apache.sentry=DEBUG</codeph> to the <filepath>log4j.properties</filepath> + file on each host in the cluster, in the appropriate configuration directory for each service. </li> </ul> Specifically, look for exceptions and messages such as: http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/77208d36/docs/topics/impala_authorization.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_authorization.xml b/docs/topics/impala_authorization.xml index 39f0e81..7890d71 100644 --- a/docs/topics/impala_authorization.xml +++ b/docs/topics/impala_authorization.xml @@ -38,10 +38,10 @@ under the License. <p> Authorization determines which users are allowed to access which resources, and what operations they are - allowed to perform. In Impala 1.1 and higher, you use the Sentry open source project for authorization. - Sentry adds a fine-grained authorization framework for Hadoop. By default (when authorization is not - enabled), Impala does all read and write operations with the privileges of the <codeph>impala</codeph> user, - which is suitable for a development/test environment but not for a secure production environment. When + allowed to perform. In Impala 1.1 and higher, you use Apache Sentry for + authorization. Sentry adds a fine-grained authorization framework for Hadoop. By default (when authorization + is not enabled), Impala does all read and write operations with the privileges of the <codeph>impala</codeph> + user, which is suitable for a development/test environment but not for a secure production environment. When authorization is enabled, Impala uses the OS user ID of the user who runs <cmdname>impala-shell</cmdname> or other client program, and associates various privileges with each user. </p> @@ -74,10 +74,11 @@ under the License. <p rev="2.3.0 collevelauth"> The object hierarchy for Impala covers Server, URI, Database, Table, and Column. (The Table privileges apply to views as well; anywhere you specify a table name, you can specify a view name instead.) - Column-level authorization is available in <keyword keyref="impala23_full"/> and higher, as described in - <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html"; format="html" scope="external"/>. - Previously, you constructed views to query specific columns and assigned privileges - based on the views rather than the base tables. + Column-level authorization is available in <keyword keyref="impala23_full"/> and higher. + Previously, you constructed views to query specific columns and assigned privilege based on + the views rather than the base tables. Now, you can use Impala's <xref href="impala_grant.xml"/> and + <xref href="impala_revoke.xml"/> statements to assign and revoke privileges from specific columns + in a table. </p> <p> @@ -234,9 +235,9 @@ under the License. <li> <p> - In an environment not managed by Cloudera Manager, you specify this value for the - <codeph>sentry.hive.server</codeph> property in the <filepath>sentry-site.xml</filepath> configuration - file for Hive, as well as in the <codeph>-server_name</codeph> option for <cmdname>impalad</cmdname>. + Specify the <codeph>server1</codeph> value for the <codeph>sentry.hive.server</codeph> property in the + <filepath>sentry-site.xml</filepath> configuration file for Hive, as well as in the + <codeph>-server_name</codeph> option for <cmdname>impalad</cmdname>. </p> <p> If the <cmdname>impalad</cmdname> daemon is not already running, start it as described in @@ -282,7 +283,7 @@ report_generator = server=server1->db=reporting_db->table=*->action=SEL <codeph>REVOKE</codeph> statements in <keyword keyref="impala20_full"/>.) </p> - <p> + <p audience="hidden"> Hive already had <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements prior to CDH 5.1, but those statements were not production-ready. CDH 5.1 is the first release where those statements use the Sentry framework and are considered GA level. If you used the Hive <codeph>GRANT</codeph> and @@ -290,10 +291,9 @@ report_generator = server=server1->db=reporting_db->table=*->action=SEL versions of <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> to take advantage of Sentry authorization. </p> - <p> + <p audience="hidden"> For information about using the updated Hive <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> statements, see -<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh_sg_sentry_service.html --> <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_sentry_service.html"; scope="external" format="html">Sentry service</xref> topic in the <cite>CDH 5 Security Guide</cite>. </p> @@ -316,10 +316,11 @@ report_generator = server=server1->db=reporting_db->table=*->action=SEL <note rev="1.4.0"> <p rev="1.4.0"> - In <ph rev="upstream">CDH 5</ph> and higher, <ph rev="upstream">Cloudera</ph> recommends - managing privileges through SQL statements, as described in - <xref href="impala_authorization.xml#sentry_service"/>. If you are still using policy files, plan to - migrate to the new approach some time in the future. + The Sentry service, as described in <xref href="impala_authorization.xml#sentry_service"/>, stores + authorization metadata in a relational database. This means you can manage user privileges for Impala tables + using traditional <codeph>GRANT</codeph> and <codeph>REVOKE</codeph> SQL statements, rather than the + policy file approach described here.If you are still using policy files, migrate to the + database-backed service whenever practical. </p> </note> @@ -504,7 +505,7 @@ entire_server = server=server1 </p> <codeblock>[groups] -cloudera = training_sysadmin, instructor +employee = training_sysadmin, instructor visitor = student [roles] @@ -588,7 +589,7 @@ student = server=server1->db=training->table=lesson_*->action=SELECT <li> The <codeph>staging_dir</codeph> role lets us specify the HDFS path - <filepath>/user/cloudera/external_data</filepath> with the <codeph>LOAD DATA</codeph> statement. + <filepath>/user/username/external_data</filepath> with the <codeph>LOAD DATA</codeph> statement. Remember, when Impala queries or loads data files, it operates on all the files in that directory, not just a single file, so any Impala <codeph>LOCATION</codeph> parameters refer to a directory rather than an individual file. @@ -605,21 +606,21 @@ student = server=server1->db=training->table=lesson_*->action=SELECT <li> We start this example after the table <codeph>external_table.sample</codeph> is already created. In the policy file for the example, we have already taken away the <codeph>external_table_admin</codeph> - role from the <codeph>cloudera</codeph> group, and replaced it with the lesser-privileged + role from the <codeph>username</codeph> group, and replaced it with the lesser-privileged <codeph>external_table</codeph> role. </li> <li> - We assign privileges to a subdirectory underneath <filepath>/user/cloudera</filepath> in HDFS, + We assign privileges to a subdirectory underneath <filepath>/user/username</filepath> in HDFS, because such privileges also apply to any subdirectories underneath. If we had assigned privileges to - the parent directory <filepath>/user/cloudera</filepath>, it would be too likely to mess up other + the parent directory <filepath>/user/username</filepath>, it would be too likely to mess up other files by specifying a wrong location by mistake. </li> <li> - The <codeph>cloudera</codeph> under the <codeph>[groups]</codeph> section refers to the - <codeph>cloudera</codeph> group. (In the demo VM used for this example, there is a - <codeph>cloudera</codeph> user that is a member of a <codeph>cloudera</codeph> group.) + The <codeph>username</codeph> under the <codeph>[groups]</codeph> section refers to the + <codeph>username</codeph> group. (In this example, there is a <codeph>username</codeph> user + that is a member of a <codeph>username</codeph> group.) </li> </ul> @@ -628,12 +629,12 @@ student = server=server1->db=training->table=lesson_*->action=SELECT </p> <codeblock>[groups] -cloudera = external_table, staging_dir +username = external_table, staging_dir [roles] external_table_admin = server=server1->db=external_table external_table = server=server1->db=external_table->table=sample->action=* -staging_dir = server=server1->uri=hdfs://127.0.0.1:8020/user/cloudera/external_data->action=* +staging_dir = server=server1->uri=hdfs://127.0.0.1:8020/user/username/external_data->action=* </codeblock> <p> @@ -664,8 +665,8 @@ Query finished, fetching results ... +-----+ Returned 3 row(s) in 1.04s -[localhost:21000] > load data inpath '/user/cloudera/external_data' into table sample; -Query: load data inpath '/user/cloudera/external_data' into table sample +[localhost:21000] > load data inpath '/user/username/external_data' into table sample; +Query: load data inpath '/user/username/external_data' into table sample Query finished, fetching results ... +----------------------------------------------------------+ | summary | @@ -691,9 +692,9 @@ Query finished, fetching results ... +-------+ Returned 9 row(s) in 0.22s -[localhost:21000] > load data inpath '/user/cloudera/unauthorized_data' into table sample; -Query: load data inpath '/user/cloudera/unauthorized_data' into table sample -ERROR: AuthorizationException: User 'cloudera' does not have privileges to access: hdfs://127.0.0.1:8020/user/cloudera/unauthorized_data +[localhost:21000] > load data inpath '/user/username/unauthorized_data' into table sample; +Query: load data inpath '/user/username/unauthorized_data' into table sample +ERROR: AuthorizationException: User 'username' does not have privileges to access: hdfs://127.0.0.1:8020/user/username/unauthorized_data </codeblock> </example> @@ -744,7 +745,7 @@ ERROR: AuthorizationException: User 'cloudera' does not have privileges to acces </p> <codeblock>[groups] -cloudera = view_only_privs +employee = view_only_privs [roles] view_only_privs = server=server1->db=reports->table=name_address_view->action=SELECT @@ -783,11 +784,10 @@ view_only_privs = server=server1->db=reports->table=name_address_view-> so can set up a database named <codeph>training</codeph>. </li> - <li> - Members of the <codeph>cloudera</codeph> group have the <codeph>instructor</codeph> role and so can - create, insert into, and query any tables in the <codeph>training</codeph> database, but cannot - create or drop the database itself. - </li> + <li> Members of the <codeph>employee</codeph> group have the + <codeph>instructor</codeph> role and so can create, insert into, + and query any tables in the <codeph>training</codeph> database, + but cannot create or drop the database itself. </li> <li> Members of the <codeph>visitor</codeph> group have the <codeph>student</codeph> role and so can query @@ -797,7 +797,7 @@ view_only_privs = server=server1->db=reports->table=name_address_view-> <codeblock>[groups] supergroup = training_sysadmin -cloudera = instructor +employee = instructor visitor = student [roles] @@ -848,8 +848,10 @@ sales = hdfs://ha-nn-uri/etc/access/sales.ini </codeblock> <p> - To enable URIs in per-DB policy files, add the following string in the Cloudera Manager field - <uicontrol>Impala Service Environment Advanced Configuration Snippet (Safety Valve)</uicontrol>: + To enable URIs in per-DB policy files, the Java configuration option <codeph>sentry.allow.uri.db.policyfile</codeph> + must be set to <codeph>true</codeph>. + For example: <!-- in the Cloudera Manager field <uicontrol>Impala Service Environment + Advanced Configuration Snippet (Safety Valve)</uicontrol>: --> </p> <codeblock>JAVA_TOOL_OPTIONS="-Dsentry.allow.uri.db.policyfile=true" @@ -931,12 +933,12 @@ Database </p> <p rev="2.3.0 collevelauth"> - In <keyword keyref="impala23_full"/> and higher, you can specify privileges for individual columns, - as described in - <xref audience="integrated" href="sg_hive_sql.xml#concept_c2q_4qx_p4/col_level_auth_sentry"/><xref audience="standalone" href="https://www.cloudera.com/documentation/enterprise/latest/topics/sg_hive_sql.html"; format="html" scope="external"/>. - Formerly, to specify - read privileges at this level, you created a view that queried specific columns and/or partitions from a base - table, and gave <codeph>SELECT</codeph> privilege on the view but not the underlying table. + In <keyword keyref="impala23_full"/> and higher, you can specify privileges for individual columns. + Formerly, to specify read privileges at this level, you created a view that queried specific columns + and/or partitions from a base table, and gave <codeph>SELECT</codeph> privilege on the view but not + the underlying table. Now, you can use Impala's <xref href="impala_grant.xml"/> and + <xref href="impala_revoke.xml"/> statements to assign and revoke privileges from specific columns + in a table. </p> <p> @@ -948,8 +950,8 @@ server=server1->uri=hdfs://namenode:port/path/to/dir </codeblock> <note type="warning"> <p> - Because the NameNode host and port must be specified, Cloudera strongly recommends you use High - Availability (HA). This ensures that the URI will remain constant even if the NameNode changes. + Because the NameNode host and port must be specified, enable High Availability (HA) to ensure + that the URI will remain constant even if the NameNode changes. </p> <codeblock>data_read = server=server1->uri=file:///path/to/dir,\ server=server1->uri=hdfs://ha-nn-uri/path/to/dir </codeblock> @@ -1020,10 +1022,6 @@ server=server1->uri=hdfs://namenode:port/path/to/dir </li> </ul> </note> - - <draft-comment author="ambreen.kazi" translate="no">The Privilege Model lives at: -https://wiki.cloudera.com/pages/viewpage.action?pageId=24919851</draft-comment> - <table> <tgroup cols="4"> <colspec colnum="1" colname="col1" colwidth="1.31*"/> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/77208d36/docs/topics/impala_security.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_security.xml b/docs/topics/impala_security.xml index 42fdaa9..a376c58 100644 --- a/docs/topics/impala_security.xml +++ b/docs/topics/impala_security.xml @@ -37,13 +37,12 @@ under the License. <conbody> <p> - Impala includes a fine-grained authorization framework for Hadoop, based on the Sentry - open source project. Sentry authorization was added in Impala 1.1.0. Together with the Kerberos authentication - framework, Sentry takes Hadoop security to a new level needed for the requirements of highly regulated industries - such as healthcare, financial services, and government. Impala also includes - an auditing capability; Impala generates the audit data, the Cloudera Navigator product consolidates - the audit data from all nodes in the cluster, and Cloudera Manager lets you filter, visualize, and produce - reports. The auditing feature was added in Impala 1.1.1. + Impala includes a fine-grained authorization framework for Hadoop, based on Apache Sentry. + Sentry authorization was added in Impala 1.1.0. Together with the Kerberos + authentication framework, Sentry takes Hadoop security to a new level needed for the requirements of + highly regulated industries such as healthcare, financial services, and government. Impala also includes + an auditing capability which was added in Impala 1.1.1; Impala generates the audit data which can be + consumed, filtered, and visualized by cluster-management components focused on governance. </p> <p> @@ -114,9 +113,9 @@ under the License. What operations were attempted, and did they succeed or not? This feature provides a way to look back and diagnose whether attempts were made to perform unauthorized operations. You use this information to track down suspicious activity, and to see where changes are needed in authorization policies. The audit data - produced by this feature is collected by the Cloudera Manager product and then presented in a - user-friendly form by the Cloudera Manager product. See <xref href="impala_auditing.xml#auditing"/> for - details about setting up and managing auditing. + produced by this feature can be collected and presented in a user-friendly form by cluster-management + software. See <xref href="impala_auditing.xml#auditing"/> for details about setting up and managing + auditing. </dd> </dlentry> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/77208d36/docs/topics/impala_security_install.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_security_install.xml b/docs/topics/impala_security_install.xml index 1bd50ad..04a5cbf 100644 --- a/docs/topics/impala_security_install.xml +++ b/docs/topics/impala_security_install.xml @@ -35,8 +35,7 @@ under the License. Impala 1.1 comes set up with all the software and settings needed to enable security when you run the <cmdname>impalad</cmdname> daemon with the new security-related options (<codeph>-server_name</codeph> and <codeph>-authorization_policy_file</codeph>). You do not need to change any environment variables or install - any additional JAR files. In a cluster managed by Cloudera Manager, you do not need to change any settings in - Cloudera Manager. + any additional JAR files. </p> </conbody> </concept> http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/77208d36/docs/topics/impala_ssl.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_ssl.xml b/docs/topics/impala_ssl.xml index 8f1e248..4e0b722 100644 --- a/docs/topics/impala_ssl.xml +++ b/docs/topics/impala_ssl.xml @@ -37,21 +37,24 @@ under the License. <p> <indexterm audience="hidden">SSL</indexterm> - Impala supports TLS/SSL network encryption, between Impala and client programs, and between the Impala-related daemons running on - different nodes in the cluster. This feature is important when you also use other features such as Kerberos authentication or Sentry - authorization, where credentials are being transmitted back and forth. - <!-- Formerly: conref="../shared/CDHVariables.xml#xd_583c10bfdbd326ba-3ca24a24-13d80143249- -7f9a/CMCDH_EitherOK" --> - <note type="important" id="CMCDH_EitherOK"> + Impala supports TLS/SSL network encryption, between Impala and client + programs, and between the Impala-related daemons running on different nodes + in the cluster. This feature is important when you also use other features such as Kerberos + authentication or Sentry authorization, where credentials are being + transmitted back and forth. + <note type="important" id="CMCDH_EitherOK" audience="hidden"> <ul id="ul_e2s_bcd_np"> - <li>You can use either Cloudera Manager or the following command-line instructions - to complete this configuration.</li> + <li>You can use either Cloudera Manager or the following command-line + instructions to complete this configuration.</li> <!-- Took out another too-specific conref, to the CDH minor version also in CDHVariables.xml. --> - <li>This information applies specifically to the version of Impala shown in the HTML page header - or on the PDF title page. - If you use an earlier version of CDH, see the documentation for that version located at - <xref href="http://www.cloudera.com/content/support/en/documentation.html"; format="html" scope="external">Cloudera Documentation</xref>.</li> - </ul></note> - /> + <li>This information applies specifically to the version of Impala + shown in the HTML page header or on the PDF title page. If you use + an earlier version of CDH, see the documentation for that version + located at <xref + href="http://www.cloudera.com/content/support/en/documentation.html"; + format="html" scope="external">Cloudera Documentation</xref>.</li> + </ul> + </note> </p> </conbody> @@ -194,11 +197,6 @@ under the License. <title>Using the Command Line</title> <conbody> - -<!-- -Info from Henry, from https://docs.google.com/a/cloudera.com/document/d/1u00CJ8WRzXR-1AK_WnQlR6LMtY-7Rc3eHaKNgw3IZvA/edit ---> - <p> To enable SSL for when client applications connect to Impala, add both of the following flags to the <cmdname>impalad</cmdname> startup options: </p> @@ -230,9 +228,8 @@ Info from Henry, from https://docs.google.com/a/cloudera.com/document/d/1u00CJ8W </p> <p> - Since Impala uses passphrase-less certificates in PEM format, you can reuse a host's existing Java keystore by converting it to the - PEM format. For instructions, see - <xref audience="integrated" href="cm_sg_openssl_jks.xml#concept_ek3_sdl_rp"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_openssl_jks.html"; scope="external" format="html"/>. + Since Impala uses passphrase-less certificates in PEM format, you can reuse a host's existing Java keystore + by using the <codeph>openssl</codeph> toolkit to convert it to the PEM format. </p> <section id="secref"> @@ -240,8 +237,7 @@ Info from Henry, from https://docs.google.com/a/cloudera.com/document/d/1u00CJ8W <title>Configuring TLS/SSL Communication for the Impala Shell</title> <p> - Typically, a client program has corresponding configuration properties in Cloudera Manager to verify that it is connecting to the - right server. For example, with SSL enabled for Impala, you use the following options when starting the + With SSL enabled for Impala, use the following options when starting the <cmdname>impala-shell</cmdname> interpreter: </p>
