[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484051#comment-17484051 ] ASF GitHub Bot commented on DRILL-8061: --- luocooong merged pull request #2422: URL: https://github.com/apache/drill/pull/2422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483780#comment-17483780
]
ASF GitHub Bot commented on DRILL-8061:
---
vdiravka commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r794507192
##
File path:
contrib/storage-phoenix/src/test/java/org/apache/drill/exec/store/phoenix/secured/QueryServerEnvironment.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix.secured;
+
+import static org.apache.hadoop.hbase.HConstants.HBASE_DIR;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.InetAddress;
+import java.security.PrivilegedAction;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.LocalHBaseCluster;
+import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.phoenix.query.ConfigurationFactory;
+import org.apache.phoenix.queryserver.QueryServerProperties;
+import org.apache.phoenix.queryserver.server.QueryServer;
+import org.apache.phoenix.util.InstanceResolver;
+import org.apache.phoenix.util.ThinClientUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Due to this bug https://bugzilla.redhat.com/show_bug.cgi?id=668830 We need
to use
+ * `localhost.localdomain` as host name when running these tests on Jenkins
(Centos) but for Mac OS
+ * it should be `localhost` to pass. The reason is kerberos principals in this
tests are looked up
+ * from /etc/hosts and a reverse DNS lookup of 127.0.0.1 is resolved to
`localhost.localdomain`
+ * rather than `localhost` on Centos. KDC sees `localhost` !=
`localhost.localdomain` and as the
+ * result test fails with authentication error. It's also important to note
these principals are
+ * shared between HDFs and HBase in this mini HBase cluster. Some more reading
+ * https://access.redhat.com/solutions/57330
+ */
+public class QueryServerEnvironment {
+ private static final Logger LOG =
LoggerFactory.getLogger(QueryServerEnvironment.class);
+
+ private final File TEMP_DIR = new File(getTempDir());
+ private final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs");
+ private final List USER_KEYTAB_FILES = new ArrayList<>();
+
+ private static final String LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ static final String LOGIN_USER;
+
+ static {
+try {
+ System.setProperty("sun.security.krb5.debug", "true");
+ LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME =
InetAddress.getByName("127.0.0.1").getCanonicalHostName();
+ String userName = System.getProperty("user.name");
+ LOGIN_USER = userName != null ? userName : "securecluster";
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ private static final String SPNEGO_PRINCIPAL = "HTTP/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String PQS_PRINCIPAL = "phoenixqs/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String SERVICE_PRINCIPAL = LOGIN_USER + "/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private File KEYTAB;
+
+ private MiniKdc KDC;
+ private HBaseTestingUtility UTIL = new HBaseTestingUtility();
+ private LocalHBaseCluster HBASE_CLUSTER;
+ private int NUM_CREATED_USERS;
+
+ private ExecutorService PQS_EXECUTOR;
+ priva
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482807#comment-17482807 ] ASF GitHub Bot commented on DRILL-8061: --- luocooong commented on pull request #2422: URL: https://github.com/apache/drill/pull/2422#issuecomment-1022718968 @vdiravka Thanks for the revision, we got better results. There are a few small suggestions that need to be worked on (marked "this one"), and we can squash all commits after the new revision is complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482804#comment-17482804
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r793150318
##
File path:
contrib/storage-phoenix/src/test/java/org/apache/drill/exec/store/phoenix/secured/QueryServerEnvironment.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix.secured;
+
+import static org.apache.hadoop.hbase.HConstants.HBASE_DIR;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.InetAddress;
+import java.security.PrivilegedAction;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.LocalHBaseCluster;
+import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.phoenix.query.ConfigurationFactory;
+import org.apache.phoenix.queryserver.QueryServerProperties;
+import org.apache.phoenix.queryserver.server.QueryServer;
+import org.apache.phoenix.util.InstanceResolver;
+import org.apache.phoenix.util.ThinClientUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Due to this bug https://bugzilla.redhat.com/show_bug.cgi?id=668830 We need
to use
+ * `localhost.localdomain` as host name when running these tests on Jenkins
(Centos) but for Mac OS
+ * it should be `localhost` to pass. The reason is kerberos principals in this
tests are looked up
+ * from /etc/hosts and a reverse DNS lookup of 127.0.0.1 is resolved to
`localhost.localdomain`
+ * rather than `localhost` on Centos. KDC sees `localhost` !=
`localhost.localdomain` and as the
+ * result test fails with authentication error. It's also important to note
these principals are
+ * shared between HDFs and HBase in this mini HBase cluster. Some more reading
+ * https://access.redhat.com/solutions/57330
+ */
+public class QueryServerEnvironment {
+ private static final Logger LOG =
LoggerFactory.getLogger(QueryServerEnvironment.class);
+
+ private final File TEMP_DIR = new File(getTempDir());
+ private final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs");
+ private final List USER_KEYTAB_FILES = new ArrayList<>();
+
+ private static final String LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ static final String LOGIN_USER;
+
+ static {
+try {
+ System.setProperty("sun.security.krb5.debug", "true");
+ LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME =
InetAddress.getByName("127.0.0.1").getCanonicalHostName();
+ String userName = System.getProperty("user.name");
+ LOGIN_USER = userName != null ? userName : "securecluster";
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ private static final String SPNEGO_PRINCIPAL = "HTTP/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String PQS_PRINCIPAL = "phoenixqs/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String SERVICE_PRINCIPAL = LOGIN_USER + "/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private File KEYTAB;
+
+ private MiniKdc KDC;
+ private HBaseTestingUtility UTIL = new HBaseTestingUtility();
+ private LocalHBaseCluster HBASE_CLUSTER;
+ private int NUM_CREATED_USERS;
+
+ private ExecutorService PQS_EXECUTOR;
+ priv
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482803#comment-17482803
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r793149883
##
File path:
contrib/storage-phoenix/src/test/java/org/apache/drill/exec/store/phoenix/secured/HttpParamImpersonationQueryServerIT.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix.secured;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
+import org.apache.hadoop.hbase.security.access.AccessControlClient;
+import org.apache.hadoop.hbase.security.access.AccessController;
+import org.apache.hadoop.hbase.security.access.Permission.Action;
+import org.apache.hadoop.hbase.security.token.TokenProvider;
+import org.apache.phoenix.end2end.NeedsOwnMiniClusterTest;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.queryserver.QueryServerOptions;
+import org.apache.phoenix.queryserver.QueryServerProperties;
+import org.apache.phoenix.queryserver.client.Driver;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static
org.apache.drill.exec.store.phoenix.secured.QueryServerEnvironment.LOGIN_USER;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class HttpParamImpersonationQueryServerIT {
+
+private static final Logger LOG =
LoggerFactory.getLogger(HttpParamImpersonationQueryServerIT.class);
+public static QueryServerEnvironment environment;
+
+private static final List SYSTEM_TABLE_NAMES = Arrays.asList(
+PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_FUNCTION_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_SCHEMA_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_SEQUENCE_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_STATS_HBASE_TABLE_NAME);
+
+public static synchronized void startQueryServerEnvironment() throws
Exception {
+//Clean up previous environment if any (Junit 4.13 @BeforeParam /
@AfterParam would be an alternative)
+if(environment != null) {
+stopEnvironment();
+}
+
+final Configuration conf = new Configuration();
+conf.setStrings(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY,
AccessController.class.getName());
+conf.setStrings(CoprocessorHost.REGIONSERVER_COPROCESSOR_CONF_KEY,
AccessController.class.getName());
+conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
AccessController.class.getName(), TokenProvider.class.getName());
+
+// Set the proxyuser settings,
+// so that the user who is running the Drillbits/MiniDfs can
impersonate user1 and user2 (not user3)
+conf.set(String.format("hadoop.proxyuser.%s.hosts", LOGIN_USER), "*");
+conf.set(String.format("hadoop.proxyuser.%s.users", LOGIN_USER),
"user1,user2");
+
conf.setBoolean(QueryServerProperties.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
true);
+environment = new QueryServerEnvironment(conf, 3, false);
+}
+
+public static synchronized void stopEnvironment() throws Exception {
+environment.stop();
+environment = null;
+}
+
+static public String getUrlTemplate() {
+String url = Driver.CONNECT_STRING_PREFIX + "url=%s://localhost:" +
environment.getPqsPort() + "?"
++
QueryServerOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM +
"=%s;authentication=SPNEGO;serialization=PROTOBUF%s";
+if (environment.getTls()) {
+return String.format(url, "https", "%s", ";truststore=" +
TlsUtil.getTrustStoreFile().getAbsolutePath()
++ ";truststore_password="+TlsUtil.getTrustStorePassword());
Review c
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480649#comment-17480649 ] ASF GitHub Bot commented on DRILL-8061: --- Z0ltrix commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r790289150 ## File path: contrib/storage-phoenix/README.md ## @@ -87,6 +79,16 @@ Tips : } ``` +### Impersonation +Configurations : +1. Enable [Drill User Impersonation](https://drill.apache.org/docs/configuring-user-impersonation/) +2. Enable [PQS Impersonation](https://phoenix.apache.org/server.html#Impersonation) +3. PQS URL: + 1. Provide `host` and `port` and Drill will generate the PQS URL with a doAs parameter of current session user + 2. Provide the `jdbcURL` with a `doAs` url param and `$user` placeholder as a value, for instance: Review comment: Ah... Sounds legit. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480647#comment-17480647 ] ASF GitHub Bot commented on DRILL-8061: --- vdiravka commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r790288210 ## File path: contrib/storage-phoenix/README.md ## @@ -87,6 +79,16 @@ Tips : } ``` +### Impersonation +Configurations : +1. Enable [Drill User Impersonation](https://drill.apache.org/docs/configuring-user-impersonation/) +2. Enable [PQS Impersonation](https://phoenix.apache.org/server.html#Impersonation) +3. PQS URL: + 1. Provide `host` and `port` and Drill will generate the PQS URL with a doAs parameter of current session user + 2. Provide the `jdbcURL` with a `doAs` url param and `$user` placeholder as a value, for instance: Review comment: Hi Christian, `doAs=$user` is adding automatically in case the first approach of configuring PQS is chosen - `host` and `port`. As Paul mentioned [earlier](https://github.com/apache/drill/pull/2332#discussion_r758116891) in the original PR #2332 for Phoenix plugin, the second approach is more advanced case and for more manual configuration. So I preferred to avoid automagic change of the final URL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480571#comment-17480571
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r790245946
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorContextImpl.java
##
@@ -98,12 +98,7 @@ public RESULT call() throws Exception {
currentThread.setName(proxyUgi.getUserName() +
":task-delegate-thread");
final RESULT result;
try {
- result = proxyUgi.doAs(new PrivilegedExceptionAction() {
-@Override
-public RESULT run() throws Exception {
- return callable.call();
Review comment:
Perhaps we should consider the users of Eclipse and IDEA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Impersonation Support for Phoenix
> -
>
> Key: DRILL-8061
> URL: https://issues.apache.org/jira/browse/DRILL-8061
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.20.0
>
>
> *Discussion:*
> * [https://github.com/apache/drill/issues/2296]
> *Documentation:*
> * [https://phoenix.apache.org/server.html#Impersonation]
> * [https://drill.apache.org/docs/configuring-user-impersonation]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480568#comment-17480568
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r790244115
##
File path:
contrib/storage-phoenix/src/test/java/org/apache/drill/exec/store/phoenix/secured/QueryServerEnvironment.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix.secured;
+
+import static org.apache.hadoop.hbase.HConstants.HBASE_DIR;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.InetAddress;
+import java.security.PrivilegedAction;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.LocalHBaseCluster;
+import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
+import org.apache.phoenix.query.ConfigurationFactory;
+import org.apache.phoenix.queryserver.QueryServerProperties;
+import org.apache.phoenix.queryserver.server.QueryServer;
+import org.apache.phoenix.util.InstanceResolver;
+import org.apache.phoenix.util.ThinClientUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Due to this bug https://bugzilla.redhat.com/show_bug.cgi?id=668830 We need
to use
+ * `localhost.localdomain` as host name when running these tests on Jenkins
(Centos) but for Mac OS
+ * it should be `localhost` to pass. The reason is kerberos principals in this
tests are looked up
+ * from /etc/hosts and a reverse DNS lookup of 127.0.0.1 is resolved to
`localhost.localdomain`
+ * rather than `localhost` on Centos. KDC sees `localhost` !=
`localhost.localdomain` and as the
+ * result test fails with authentication error. It's also important to note
these principals are
+ * shared between HDFs and HBase in this mini HBase cluster. Some more reading
+ * https://access.redhat.com/solutions/57330
+ */
+public class QueryServerEnvironment {
+ private static final Logger LOG =
LoggerFactory.getLogger(QueryServerEnvironment.class);
+
+ private final File TEMP_DIR = new File(getTempDir());
+ private final File KEYTAB_DIR = new File(TEMP_DIR, "keytabs");
+ private final List USER_KEYTAB_FILES = new ArrayList<>();
+
+ private static final String LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ static final String LOGIN_USER;
+
+ static {
+try {
+ System.setProperty("sun.security.krb5.debug", "true");
+ LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME =
InetAddress.getByName("127.0.0.1").getCanonicalHostName();
+ String userName = System.getProperty("user.name");
+ LOGIN_USER = userName != null ? userName : "securecluster";
+} catch (Exception e) {
+ throw new RuntimeException(e);
+}
+ }
+
+ private static final String SPNEGO_PRINCIPAL = "HTTP/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String PQS_PRINCIPAL = "phoenixqs/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private static final String SERVICE_PRINCIPAL = LOGIN_USER + "/" +
LOCAL_HOST_REVERSE_DNS_LOOKUP_NAME;
+ private File KEYTAB;
+
+ private MiniKdc KDC;
+ private HBaseTestingUtility UTIL = new HBaseTestingUtility();
+ private LocalHBaseCluster HBASE_CLUSTER;
+ private int NUM_CREATED_USERS;
+
+ private ExecutorService PQS_EXECUTOR;
+ priv
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480567#comment-17480567 ] ASF GitHub Bot commented on DRILL-8061: --- Z0ltrix commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r787684404 ## File path: contrib/storage-phoenix/README.md ## @@ -87,6 +79,16 @@ Tips : } ``` +### Impersonation +Configurations : +1. Enable [Drill User Impersonation](https://drill.apache.org/docs/configuring-user-impersonation/) +2. Enable [PQS Impersonation](https://phoenix.apache.org/server.html#Impersonation) +3. PQS URL: + 1. Provide `host` and `port` and Drill will generate the PQS URL with a doAs parameter of current session user + 2. Provide the `jdbcURL` with a `doAs` url param and `$user` placeholder as a value, for instance: Review comment: wouldnt it be better, if drill would add the doAs=$user automatically if impersonation is enabled? It is confusing, If you have to change the storage config AND drill-override.conf to enable impersonation without an error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480566#comment-17480566
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r790242245
##
File path:
contrib/storage-phoenix/src/test/java/org/apache/drill/exec/store/phoenix/secured/HttpParamImpersonationQueryServerIT.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix.secured;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
+import org.apache.hadoop.hbase.security.access.AccessControlClient;
+import org.apache.hadoop.hbase.security.access.AccessController;
+import org.apache.hadoop.hbase.security.access.Permission.Action;
+import org.apache.hadoop.hbase.security.token.TokenProvider;
+import org.apache.phoenix.end2end.NeedsOwnMiniClusterTest;
+import org.apache.phoenix.jdbc.PhoenixDatabaseMetaData;
+import org.apache.phoenix.queryserver.QueryServerOptions;
+import org.apache.phoenix.queryserver.QueryServerProperties;
+import org.apache.phoenix.queryserver.client.Driver;
+import org.junit.experimental.categories.Category;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static
org.apache.drill.exec.store.phoenix.secured.QueryServerEnvironment.LOGIN_USER;
+
+@Category(NeedsOwnMiniClusterTest.class)
+public class HttpParamImpersonationQueryServerIT {
+
+private static final Logger LOG =
LoggerFactory.getLogger(HttpParamImpersonationQueryServerIT.class);
+public static QueryServerEnvironment environment;
+
+private static final List SYSTEM_TABLE_NAMES = Arrays.asList(
+PhoenixDatabaseMetaData.SYSTEM_CATALOG_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_MUTEX_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_FUNCTION_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_SCHEMA_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_SEQUENCE_HBASE_TABLE_NAME,
+PhoenixDatabaseMetaData.SYSTEM_STATS_HBASE_TABLE_NAME);
+
+public static synchronized void startQueryServerEnvironment() throws
Exception {
+//Clean up previous environment if any (Junit 4.13 @BeforeParam /
@AfterParam would be an alternative)
+if(environment != null) {
+stopEnvironment();
+}
+
+final Configuration conf = new Configuration();
+conf.setStrings(CoprocessorHost.MASTER_COPROCESSOR_CONF_KEY,
AccessController.class.getName());
+conf.setStrings(CoprocessorHost.REGIONSERVER_COPROCESSOR_CONF_KEY,
AccessController.class.getName());
+conf.setStrings(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
AccessController.class.getName(), TokenProvider.class.getName());
+
+// Set the proxyuser settings,
+// so that the user who is running the Drillbits/MiniDfs can
impersonate user1 and user2 (not user3)
+conf.set(String.format("hadoop.proxyuser.%s.hosts", LOGIN_USER), "*");
+conf.set(String.format("hadoop.proxyuser.%s.users", LOGIN_USER),
"user1,user2");
+
conf.setBoolean(QueryServerProperties.QUERY_SERVER_WITH_REMOTEUSEREXTRACTOR_ATTRIB,
true);
+environment = new QueryServerEnvironment(conf, 3, false);
+}
+
+public static synchronized void stopEnvironment() throws Exception {
+environment.stop();
+environment = null;
+}
+
+static public String getUrlTemplate() {
+String url = Driver.CONNECT_STRING_PREFIX + "url=%s://localhost:" +
environment.getPqsPort() + "?"
++
QueryServerOptions.DEFAULT_QUERY_SERVER_REMOTEUSEREXTRACTOR_PARAM +
"=%s;authentication=SPNEGO;serialization=PROTOBUF%s";
+if (environment.getTls()) {
+return String.format(url, "https", "%s", ";truststore=" +
TlsUtil.getTrustStoreFile().getAbsolutePath()
++ ";truststore_password="+TlsUtil.getTrustStorePassword());
Review c
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480408#comment-17480408 ] ASF GitHub Bot commented on DRILL-8061: --- luocooong commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r790134758 ## File path: contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixDataSource.java ## @@ -21,13 +21,14 @@ import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; -import java.sql.SQLFeatureNotSupportedException; import java.util.Map; import java.util.Properties; import java.util.logging.Logger; import javax.sql.DataSource; +import lombok.extern.slf4j.Slf4j; Review comment: It seems that we have the same understanding of the situation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480406#comment-17480406
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r790134460
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -71,58 +74,64 @@ public PhoenixBatchReader(PhoenixSubScan subScan) {
@Override
public boolean open(SchemaNegotiator negotiator) {
-try {
+UserGroupInformation ugi = ImpersonationUtil.getProcessUserUGI();
+return ugi.doAs((PrivilegedAction) () -> {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
- results = pstmt.executeQuery();
- meta = pstmt.getMetaData();
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-try {
- negotiator.tableSchema(defineMetadata(), true);
- reader = new PhoenixReader(negotiator.build(), columns, results);
- bindColumns(reader.getStorage());
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to get type of columns from metadata. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-watch = Stopwatch.createStarted();
-return true;
+ try {
+pstmt =
+
subScan.getPlugin().getDataSource(negotiator.userName()).getConnection().prepareStatement(subScan.getSql());
+results = pstmt.executeQuery();
+meta = pstmt.getMetaData();
+ } catch (SQLException e) {
+throw UserException
+ .dataReadError(e)
+ .message("Failed to execute the phoenix sql query. " +
e.getMessage())
+ .addContext(negotiator.parentErrorContext())
+ .build(logger);
+ }
+ try {
+negotiator.tableSchema(defineMetadata(), true);
+reader = new PhoenixReader(negotiator.build(), columns, results);
+bindColumns(reader.getStorage());
+ } catch (SQLException e) {
+throw UserException
+ .dataReadError(e)
+ .message("Failed to get type of columns from metadata. " +
e.getMessage())
+ .addContext(errorContext)
+ .build(logger);
+ }
+ watch = Stopwatch.createStarted();
+ return true;
+});
}
@Override
public boolean next() {
+UserGroupInformation ugi = ImpersonationUtil.getProcessUserUGI();
try {
- while(!reader.getStorage().isFull()) {
-if (!reader.processRow()) { // return true if one row is processed.
- watch.stop();
- logger.debug("Phoenix fetch total record numbers : {}",
reader.getRowCount());
- return false; // the EOF is reached.
+ return ugi.doAs((PrivilegedExceptionAction) () -> {
Review comment:
I fully agree that we will not use the impersonation feature until it is
enabled, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Impersonation Support for Phoenix
> -
>
> Key: DRILL-8061
> URL: https://issues.apache.org/jira/browse/DRILL-8061
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.20.0
>
>
> *Discussion:*
> * [https://github.com/apache/drill/issues/2296]
> *Documentation:*
> * [https://phoenix.apache.org/server.html#Impersonation]
> * [https://drill.apache.org/docs/configuring-user-impersonation]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479256#comment-17479256 ] ASF GitHub Bot commented on DRILL-8061: --- jnturton commented on pull request #2422: URL: https://github.com/apache/drill/pull/2422#issuecomment-1017358574 Hi @luocooong! Has @vdiravka addressed everything with this latest push? Excuse my nudging but this is one of two open PRs for 1.20. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478627#comment-17478627
]
ASF GitHub Bot commented on DRILL-8061:
---
jnturton commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r787682748
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -67,42 +74,55 @@
public PhoenixBatchReader(PhoenixSubScan subScan) {
this.subScan = subScan;
+this.impersonationEnabled =
subScan.getPlugin().getContext().getConfig().getBoolean(ExecConstants.IMPERSONATION_ENABLED);
}
@Override
public boolean open(SchemaNegotiator negotiator) {
+return impersonationEnabled
+ ? ugi.doAs((PrivilegedAction) () -> processOpen(negotiator))
+ : processOpen(negotiator);
+ }
+
+ private boolean processOpen(SchemaNegotiator negotiator) {
try {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
+ DataSource ds = subScan.getPlugin().getDataSource(negotiator.userName());
+ PreparedStatement pstmt =
ds.getConnection().prepareStatement(subScan.getSql());
results = pstmt.executeQuery();
meta = pstmt.getMetaData();
} catch (SQLException e) {
throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
+.dataReadError(e)
+.message("Failed to execute the phoenix sql query. " + e.getMessage())
+.addContext(errorContext)
+.build(logger);
}
Review comment:
```suggestion
} finally {
pstmt.close();
}
```
Something flagged by LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Impersonation Support for Phoenix
> -
>
> Key: DRILL-8061
> URL: https://issues.apache.org/jira/browse/DRILL-8061
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.20.0
>
>
> *Discussion:*
> * [https://github.com/apache/drill/issues/2296]
> *Documentation:*
> * [https://phoenix.apache.org/server.html#Impersonation]
> * [https://drill.apache.org/docs/configuring-user-impersonation]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478626#comment-17478626
]
ASF GitHub Bot commented on DRILL-8061:
---
jnturton commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r787682748
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -67,42 +74,55 @@
public PhoenixBatchReader(PhoenixSubScan subScan) {
this.subScan = subScan;
+this.impersonationEnabled =
subScan.getPlugin().getContext().getConfig().getBoolean(ExecConstants.IMPERSONATION_ENABLED);
}
@Override
public boolean open(SchemaNegotiator negotiator) {
+return impersonationEnabled
+ ? ugi.doAs((PrivilegedAction) () -> processOpen(negotiator))
+ : processOpen(negotiator);
+ }
+
+ private boolean processOpen(SchemaNegotiator negotiator) {
try {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
+ DataSource ds = subScan.getPlugin().getDataSource(negotiator.userName());
+ PreparedStatement pstmt =
ds.getConnection().prepareStatement(subScan.getSql());
results = pstmt.executeQuery();
meta = pstmt.getMetaData();
} catch (SQLException e) {
throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
+.dataReadError(e)
+.message("Failed to execute the phoenix sql query. " + e.getMessage())
+.addContext(errorContext)
+.build(logger);
}
Review comment:
```suggestion
} finally {
pstmt.close();
```
Something flagged by LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Impersonation Support for Phoenix
> -
>
> Key: DRILL-8061
> URL: https://issues.apache.org/jira/browse/DRILL-8061
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: 1.20.0
>
>
> *Discussion:*
> * [https://github.com/apache/drill/issues/2296]
> *Documentation:*
> * [https://phoenix.apache.org/server.html#Impersonation]
> * [https://drill.apache.org/docs/configuring-user-impersonation]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478623#comment-17478623 ] ASF GitHub Bot commented on DRILL-8061: --- lgtm-com[bot] commented on pull request #2422: URL: https://github.com/apache/drill/pull/2422#issuecomment-1016404261 This pull request **introduces 1 alert** when merging 2a2e37ca73f127731cf849bcd8bce006f5c8c7a0 into ee874af953ed99614c8e7291ea9164b4ddc5ff24 - [view on LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-9217048ea766a8f5841fc4cdae019a262ea5c758) **new alerts:** * 1 for Potential database resource leak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478611#comment-17478611 ] ASF GitHub Bot commented on DRILL-8061: --- jnturton commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r787670219 ## File path: contrib/storage-phoenix/pom.xml ## @@ -33,6 +33,7 @@ 5.1.2 2.4.2 +9.4.31.v20200723 Review comment: Nevermind, my search was just broken! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478592#comment-17478592 ] ASF GitHub Bot commented on DRILL-8061: --- jnturton commented on a change in pull request #2422: URL: https://github.com/apache/drill/pull/2422#discussion_r787651312 ## File path: contrib/storage-phoenix/pom.xml ## @@ -33,6 +33,7 @@ 5.1.2 2.4.2 +9.4.31.v20200723 Review comment: @vdiravka I cannot find any pom file in the Drill tree that actually _uses_ the property jetty.version. Or does it get used in the pom files of some of Drill's dependencies? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478564#comment-17478564
]
ASF GitHub Bot commented on DRILL-8061:
---
vdiravka commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r785419426
##
File path: contrib/storage-phoenix/pom.xml
##
@@ -33,6 +33,7 @@
5.1.2
2.4.2
+9.4.31.v20200723
Review comment:
Jetty is used by testing PQS:
https://github.com/apache/phoenix-queryserver/blob/master/pom.xml#L83
Tests failed with Drill Jetty `9.4.41.v20210516` version
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -71,58 +74,64 @@ public PhoenixBatchReader(PhoenixSubScan subScan) {
@Override
public boolean open(SchemaNegotiator negotiator) {
-try {
+UserGroupInformation ugi = ImpersonationUtil.getProcessUserUGI();
+return ugi.doAs((PrivilegedAction) () -> {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
- results = pstmt.executeQuery();
- meta = pstmt.getMetaData();
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-try {
- negotiator.tableSchema(defineMetadata(), true);
- reader = new PhoenixReader(negotiator.build(), columns, results);
- bindColumns(reader.getStorage());
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to get type of columns from metadata. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-watch = Stopwatch.createStarted();
-return true;
+ try {
+pstmt =
+
subScan.getPlugin().getDataSource(negotiator.userName()).getConnection().prepareStatement(subScan.getSql());
Review comment:
ok
##
File path: contrib/storage-phoenix/pom.xml
##
@@ -187,12 +194,79 @@
${hbase.minicluster.version}
test
+
+ org.apache.hbase
+ hbase-asyncfs
+ test-jar
+ ${hbase.minicluster.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
org.apache.hadoop
hadoop-hdfs-client
${hadoop.version}
test
+
+ org.apache.hadoop
+ hadoop-minikdc
+ ${hadoop.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
+
+ org.apache.hbase
+ hbase-testing-util
+ ${hbase.minicluster.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
+
+ org.apache.hbase
+ hbase-protocol-shaded
Review comment:
Agreed. It is already declared. Thanks
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -71,58 +74,64 @@ public PhoenixBatchReader(PhoenixSubScan subScan) {
@Override
public boolean open(SchemaNegotiator negotiator) {
-try {
+UserGroupInformation ugi = ImpersonationUtil.getProcessUserUGI();
+return ugi.doAs((PrivilegedAction) () -> {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
- results = pstmt.executeQuery();
- meta = pstmt.getMetaData();
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-try {
- negotiator.tableSchema(defineMetadata(), true);
- reader = new PhoenixReader(negotiator.build(), columns, results);
- bindColumns(reader.getStorage());
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to get type of columns from metadata. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-watch = Stopwatch.createStarted();
-return true;
+
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[
https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471806#comment-17471806
]
ASF GitHub Bot commented on DRILL-8061:
---
luocooong commented on a change in pull request #2422:
URL: https://github.com/apache/drill/pull/2422#discussion_r780727669
##
File path: contrib/storage-phoenix/README.md
##
@@ -150,4 +155,13 @@ apache drill (phoenix123.v1)> select n_name, n_regionkey
from nation limit 3 off
| JAPAN | 2 |
++-+
3 rows selected (0.77 seconds)
-```
\ No newline at end of file
+```
+### Impersonation
Review comment:
Is it possible to put this section after the `Configuration`?
##
File path: contrib/storage-phoenix/pom.xml
##
@@ -33,6 +33,7 @@
5.1.2
2.4.2
+9.4.31.v20200723
Review comment:
If we do not actually use Jetty, can be removed.
##
File path: contrib/storage-phoenix/pom.xml
##
@@ -187,12 +194,79 @@
${hbase.minicluster.version}
test
+
+ org.apache.hbase
+ hbase-asyncfs
+ test-jar
+ ${hbase.minicluster.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
org.apache.hadoop
hadoop-hdfs-client
${hadoop.version}
test
+
+ org.apache.hadoop
+ hadoop-minikdc
+ ${hadoop.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
+
+ org.apache.hbase
+ hbase-testing-util
+ ${hbase.minicluster.version}
+ test
+
+
+ commons-logging
+ commons-logging
+
+
+ log4j
+ log4j
+
+
+ org.slf4j
+ *
+
+
+
+
+ org.apache.hbase
+ hbase-protocol-shaded
Review comment:
Duplicate declaration of dependencies, at line 266.
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixDataSource.java
##
@@ -21,13 +21,14 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLFeatureNotSupportedException;
import java.util.Map;
import java.util.Properties;
import java.util.logging.Logger;
import javax.sql.DataSource;
+import lombok.extern.slf4j.Slf4j;
Review comment:
I have no objection to using lombok..
But the usability that depends on the IDE plugin(Not JDK built-in) is
difficult to accept (to me personally).
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixBatchReader.java
##
@@ -71,58 +74,64 @@ public PhoenixBatchReader(PhoenixSubScan subScan) {
@Override
public boolean open(SchemaNegotiator negotiator) {
-try {
+UserGroupInformation ugi = ImpersonationUtil.getProcessUserUGI();
+return ugi.doAs((PrivilegedAction) () -> {
errorContext = negotiator.parentErrorContext();
- conn = subScan.getPlugin().getDataSource().getConnection();
- pstmt = conn.prepareStatement(subScan.getSql());
- results = pstmt.executeQuery();
- meta = pstmt.getMetaData();
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to execute the phoenix sql query. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-try {
- negotiator.tableSchema(defineMetadata(), true);
- reader = new PhoenixReader(negotiator.build(), columns, results);
- bindColumns(reader.getStorage());
-} catch (SQLException e) {
- throw UserException
- .dataReadError(e)
- .message("Failed to get type of columns from metadata. " +
e.getMessage())
- .addContext(errorContext)
- .build(logger);
-}
-watch = Stopwatch.createStarted();
-return true;
+ try {
+pstmt =
+
subScan.getPlugin().getDataSource(negotiator.userName()).getConnection().prepareStatement(subScan.getSql());
+results = pstmt.executeQuery();
+meta = pstmt.getMetaData();
+ } catch (SQLException e) {
+throw UserException
+ .dataReadError(e)
+ .message("Failed to execute the phoenix sql query. " +
e.getMessage())
+ .addContext(negotiator.parentErrorContext())
+ .build(logger);
+ }
+ try {
+negotiator.tableSchema(defineMetadata(), true);
+reader = new PhoenixReader(negotiator.build(), columns, results);
+bindColumns(reader.getS
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469030#comment-17469030 ] ASF GitHub Bot commented on DRILL-8061: --- lgtm-com[bot] commented on pull request #2422: URL: https://github.com/apache/drill/pull/2422#issuecomment-1005407994 This pull request **introduces 1 alert** when merging 5040d117a2515677df6d4cf2c7c174edb8994a89 into fa2cb0f4937c0d8e797a675d8d6c13c316e48d4c - [view on LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-18b01353d3c800d75945d94dc316664ad25dc4f4) **new alerts:** * 1 for Potential database resource leak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469006#comment-17469006 ] ASF GitHub Bot commented on DRILL-8061: --- vdiravka opened a new pull request #2422: URL: https://github.com/apache/drill/pull/2422 # [DRILL-8061](https://issues.apache.org/jira/browse/DRILL-8061): Add Impersonation Support for Phoenix ## Description Drill Hadoop User Impersonation for Apache Phoenix ## Documentation ### Impersonation Configurations : 1. Enable [Drill User Impersonation](https://drill.apache.org/docs/configuring-user-impersonation/) 2. Enable [PQS Impersonation](https://phoenix.apache.org/server.html#Impersonation) 3. PQS URL: 1. Provide `host` and `port` and Drill will generate the PQS URL with a doAs parameter of current session user 2. Provide the `jdbcURL` with a `doAs` url param and `$user` placeholder as a value, for instance: `jdbc:phoenix:thin:url=http://localhost:8765?doAs=$user`. In case Drill Impersonation is enabled, but `doAs=$user` is missing the User Exception is thrown. ## Testing Added IT test cases for the kerberized `hadoop-minikdc` Phoenix + Drill cluster. Tests run for 3 clients: * the first has all permission * Impersonated, but don't have an permissions to access the tables * not impersonated by admin Draft until all test cases pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > *Discussion:* > * [https://github.com/apache/drill/issues/2296] > *Documentation:* > * [https://phoenix.apache.org/server.html#Impersonation] > * [https://drill.apache.org/docs/configuring-user-impersonation] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453084#comment-17453084 ] Cong Luo commented on DRILL-8061: - [~vitalii] Hi, thank you for doing this. > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Vitalii Diravka >Priority: Major > Fix For: 1.20.0 > > > * https://phoenix.apache.org/server.html#Impersonation > * https://drill.apache.org/docs/configuring-user-impersonation -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-8061) Add Impersonation Support for Phoenix
[ https://issues.apache.org/jira/browse/DRILL-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452974#comment-17452974 ] Vitalii Diravka commented on DRILL-8061: Hi [~luoc] . Do you have any progress on it? Or I can take it? > Add Impersonation Support for Phoenix > - > > Key: DRILL-8061 > URL: https://issues.apache.org/jira/browse/DRILL-8061 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > Fix For: 1.20.0 > > > * https://phoenix.apache.org/server.html#Impersonation > * https://drill.apache.org/docs/configuring-user-impersonation -- This message was sent by Atlassian Jira (v8.20.1#820001)
