-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62411/
-----------------------------------------------------------
Review request for sentry, Alexander Kolbasov, Colm O hEigeartaigh, and kalyan
kumar kalvagadda.
Bugs: sentry-1958
https://issues.apache.org/jira/browse/sentry-1958
Repository: sentry
Description
-------
This patch bumps the Hive version of hive-authz1 to Hive 2.0. Moving to authz2
has been a little complicated, so doing an incremental patch was preffered.
Also, we're taking advantage of the unit tests nad e2e tests that already exist
on Sentry. There are tests that are not on the authz2, so this is why I
preffered to look into bumping the Hive 2.0 version first.
The following issues with Hive 2.0 were found and addressed on Sentry:
- Hive 2.0.1 has a bug that Sentry cannot workaround.
- Hive 2.1.1 and higher has a different bug that Sentry cannot workaround.
- Hive CBO has a bug where ReadIdentity partitions do not have parents causing
Sentry to request extra privileges that the user might not have
CBO is disabled on the Sentry tests and it must be disabled on production as
well.
- HIVE-11145: Remove OFFLINE and NO_DROP from tables and partitions
Removed tests that use the protection mode operations as Hive do not support
them any more.
- HIVE-10453: HS2 leaking open file descriptors when using UDFs
Hive 2.0 clears all functions after a session is closed causing other users
who want to execute such
function to fail because they lost the function scope and they do not have
permissions to create functions
- HIVE-12320: hive.metastore.disallow.incompatible.col.type.changes should be
true by default
Sentry had some issues on the tests due to this Hive change.
- HIVE-10307: Support to use number literals in partition column
Hive 2.0 added an extra validation when using ALTER TABLE ... PARTITION
(spec) that throws an error if
spec is not a partition column.
- The HS2 webui fails to start when concurrency mode is enabled. The Sentry
tests are now configured to put
Hive in test mode so that the webui is not initialized.
- There are some column names that cannot be used as they are reservered by
Hive, i.e 'date' column name fails
in some Sentry tests.
- Hive 2.0 switched to log4j2 causing some Sentry tests to fail.
Important changes on Sentry:
- Hive 2.0 has an authz1 bug with the use of SentryMetastoreFilterHook class.
This class is replaced automatically by Hive with a default one that uses
authz2. To make minimal changes on Sentry, a new class is created
(SentryHiveAuthorizerImpl) that only deals with metastore filtering. Also, the
class SentryHiveAuthorizerFactory is set by the HiveAuthzBindingSessionHook
automatically when a HS2 session is opened. However, this new authorizer must
be set manually on the hive-site.xml so that other clients who use the HMS
Client can use this new filter class (info is mentioned on the
SentryHiveAuthorizerFactory class.
There will be follow-up patches to fix the following:
- Remove SentryMetastoreFilterHook and improve the SentryHiveAuthorizerImpl to
avoid the overhead of converting a list of hive privileges to a list of strings
and viceversa.
- Merge authz1 and authz2 dependencies.
- Review current authz2 and merge code into the new changes.
Diffs
-----
dev-support/test-patch.py ac91b590c93884911d27928ac60108dcaa5d81ed
pom.xml 1479f5fa3895833a1efa63d0f4a7c3db72c7cde9
sentry-binding/sentry-binding-hive-follower-v2/pom.xml
fa7e9282592bf5fe15a35921b7173c81b45a1f1f
sentry-binding/sentry-binding-hive-follower/pom.xml
e69519cc63d5c89db6098fa169089b34dbaadae8
sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONAlterPartitionMessage.java
1e636c94afc99678658bdafe74fdd7aff7e12d92
sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONDropPartitionMessage.java
d3ebf603f2d81b3fed7ec0d33031446185b80cb5
sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageDeserializer.java
cc0bbec830436241ef69fc3dc9a727bf105ae6e7
sentry-binding/sentry-binding-hive-follower/src/main/java/org/apache/sentry/binding/metastore/messaging/json/SentryJSONMessageFactory.java
efdf8b8a5b4eeca08e2a4ad39c95ab86e7855a76
sentry-binding/sentry-binding-hive-v2/pom.xml
5f5cbf3c15df324d741e5eca7b951e4aab5bd0d3
sentry-binding/sentry-binding-hive/pom.xml
92147e18b0560bd266bfc82d65b9bc70af3df627
sentry-binding/sentry-binding-hive/src/main/java/org/apache/hadoop/hive/ql/exec/SentryGrantRevokeTask.java
97dbd2a503bf9d61c18f6f4fcaa494cbd429f07d
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/HiveAuthzBindingSessionHook.java
a3aa0b0351deb7b41dec3e5bc7b0e62727e3cb21
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/HiveAuthzPrivilegesMap.java
9f3d42d3787fbceb62e9fd5e22395c17010867f4
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/SentryHiveAuthorizerFactory.java
PRE-CREATION
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/hive/authz/SentryHiveAuthorizerImpl.java
PRE-CREATION
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/AuthorizingObjectStore.java
d20da81efcbc796456c114654df0261964c2cfc1
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/AuthorizingObjectStoreBase.java
9e066e12f1b3539e8787878723eec153dfd68d32
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/MetastoreAuthzBindingBase.java
6df4885cea3868b451ed14a5e354dd45794cbb3f
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListener.java
3ec2eedab99e7ec66e2dc69b9c8af8eb482cac08
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListenerBase.java
5b9274e337d1f8a05e2b01384cb3a2e95ad07293
sentry-binding/sentry-binding-hive/src/main/java/org/apache/sentry/binding/metastore/SentryMetastorePostEventListenerNotificationLog.java
58470d6ade3467ef7b03428272d61dc0dd1accea
sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestSentryHiveAuthorizationTaskFactory.java
aed218ec0308405c32e6a070451ef38c4c67bb27
sentry-binding/sentry-binding-hive/src/test/java/org/apache/sentry/binding/hive/TestURI.java
aa3de64a404b2a8e1dbe245a9cd2298c41745fc3
sentry-binding/sentry-binding-hive/src/test/resources/log4j2.properties
PRE-CREATION
sentry-provider/sentry-provider-db/pom.xml
6b7d3c07a0cdf43ff925c22d58c290a3a9affe8b
sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/service/thrift/SentryPolicyStoreProcessor.java
cd854006f2b0e73a8a1bcb3c87a4df01bd30cc14
sentry-tests/sentry-tests-hive/pom.xml
51801dc8bdcd449f9acaa5658855a4227c47c56e
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/dbprovider/TestDbDDLAuditLog.java
e105f003b79b0ad9bec58b24d3e7fd87576e7ad7
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationAdvanced.java
2073d85f27349c933bd936f87dc754c6fb30a729
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationBase.java
718b02cbd9df9a551bd4de1768f93a914cad7b32
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java
d4bc97d2f59efee00034036ebc716ab69cf2116b
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/AbstractTestWithStaticConfiguration.java
0a39f5954de5867b4e64e9be3529219654d31378
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestMetadataObjectRetrieval.java
fb0ef19a52375126d21a4c787dd31a1deacfc232
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestOperationsPart1.java
3a4da5059463459af43fd04ed3b34b6468848e7e
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/TestPrivilegesAtFunctionScope.java
249d3bcfe8b1f9f2a1256280cf9b80527cef1e4c
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hive/hiveserver/HiveServerFactory.java
f3f58f64f66ea693ab99d2dd5a9b0eb298ae7d0f
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestDBNotificationListenerInBuiltDeserializer.java
e9b3a43debb6d47818c54f6a1f646f13668e35f2
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/metastore/TestSentryListenerSentryDeserializer.java
f43b316928a6354674e824d04b8e7c4a9e5bec00
sentry-tests/sentry-tests-hive/src/test/resources/log4j2.properties
PRE-CREATION
sentry-tests/sentry-tests-kafka/pom.xml
56a3ef10a9071929776cb7211bdb8ead921deace
sentry-tests/sentry-tests-solr/pom.xml
c70476808688c80e1723d5e65e3b8cf6d1b64250
sentry-tests/sentry-tests-sqoop/pom.xml
9e7cee95be06c68b0eaa456a80d4120576ec977b
Diff: https://reviews.apache.org/r/62411/diff/1/
Testing
-------
Thanks,
Sergio Pena
