[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17453502#comment-17453502 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong merged pull request #2332: URL: https://github.com/apache/drill/pull/2332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452488#comment-17452488 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on a change in pull request #2332: URL: https://github.com/apache/drill/pull/2332#discussion_r761251990 ## File path: contrib/storage-phoenix/README.md ## @@ -0,0 +1,98 @@ +# [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix + +## Description + + Phoenix say : "We put the SQL back in NoSQL", + Drill call : "We use the SQL to cross almost all the file systems and storage engines", + "Cheers !", users said. + +## Documentation + Review comment: @luocooong Thank you so much for this submission. I don't think I could add much more than @paul-rogers did for the code review. I do have some suggestions for the docs however. Could you please add a section about how to configure Drill to connect to Phoenix and perhaps a demo config? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452478#comment-17452478 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on a change in pull request #2332: URL: https://github.com/apache/drill/pull/2332#discussion_r761238385 ## File path: contrib/storage-phoenix/README.md ## @@ -0,0 +1,98 @@ +# [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix + +## Description + + Phoenix say : "We put the SQL back in NoSQL", + Drill call : "We use the SQL to cross almost all the file systems and storage engines", + "Cheers !", users said. + +## Documentation + +Features : + + - Full support for Enhanced Vector Framework. + + - Tested in phoenix 4.14 and 5.1.2. + + - Support the array data type. + + - Support the pushdown (Project, Limit, Filter, Aggregate, Join, CrossJoin, Join_Filter, GroupBy, Distinct and more). + + - Use the PQS client (6.0). + +Related Information : + + 1. PHOENIX-6398: Returns uniform SQL dialect in calcite for the PQS + + 2. PHOENIX-6582: Bump default HBase version to 2.3.7 and 2.4.8 + + 3. PHOENIX-6605, PHOENIX-6606 and PHOENIX-6607. + + 4. DRILL-8060, DRILL-8061 and DRILL-8062. + + 5. [QueryServer 6.0.0-drill-r1](https://github.com/luocooong/phoenix-queryserver/releases/tag/6.0.0-drill-r1) + +## Testing + + The test framework of phoenix queryserver required the Hadoop 3, but exist `PHOENIX-5993` and `HBASE-22394` : + +``` +" The HBase PMC does not release multiple artifacts for both Hadoop2 and Hadoop3 support at the current time. +Current HBase2 releases still compile against Hadoop2 by default, and using Hadoop 3 against HBase2 +requires a recompilation of HBase because of incompatible changes between Hadoop2 and Hadoop3. " +``` + +### Recommended Practices + + 1. Download HBase 2.4.2 sources and rebuild with Hadoop 3. + + 2. Remove the `Ignore` annotation in `PhoenixTestSuite.java`. + + 3. Go to the phoenix root folder and run test. + +### To Add Features + + - Don't forget to add a test function to the test class. + + - If a new test class is added, please declare it in the `PhoenixTestSuite` class. + +### Play in CLI + +```sql Review comment: The `sql` keyword id fine. Using a lot of tables in the docs can throw things off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452220#comment-17452220 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on a change in pull request #2332: URL: https://github.com/apache/drill/pull/2332#discussion_r760832203 ## File path: contrib/storage-phoenix/README.md ## @@ -0,0 +1,98 @@ +# [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix + +## Description + + Phoenix say : "We put the SQL back in NoSQL", + Drill call : "We use the SQL to cross almost all the file systems and storage engines", + "Cheers !", users said. + +## Documentation + +Features : + + - Full support for Enhanced Vector Framework. + + - Tested in phoenix 4.14 and 5.1.2. + + - Support the array data type. + + - Support the pushdown (Project, Limit, Filter, Aggregate, Join, CrossJoin, Join_Filter, GroupBy, Distinct and more). + + - Use the PQS client (6.0). + +Related Information : + + 1. PHOENIX-6398: Returns uniform SQL dialect in calcite for the PQS + + 2. PHOENIX-6582: Bump default HBase version to 2.3.7 and 2.4.8 + + 3. PHOENIX-6605, PHOENIX-6606 and PHOENIX-6607. + + 4. DRILL-8060, DRILL-8061 and DRILL-8062. + + 5. [QueryServer 6.0.0-drill-r1](https://github.com/luocooong/phoenix-queryserver/releases/tag/6.0.0-drill-r1) + +## Testing + + The test framework of phoenix queryserver required the Hadoop 3, but exist `PHOENIX-5993` and `HBASE-22394` : + +``` +" The HBase PMC does not release multiple artifacts for both Hadoop2 and Hadoop3 support at the current time. +Current HBase2 releases still compile against Hadoop2 by default, and using Hadoop 3 against HBase2 +requires a recompilation of HBase because of incompatible changes between Hadoop2 and Hadoop3. " +``` + +### Recommended Practices + + 1. Download HBase 2.4.2 sources and rebuild with Hadoop 3. + + 2. Remove the `Ignore` annotation in `PhoenixTestSuite.java`. + + 3. Go to the phoenix root folder and run test. + +### To Add Features + + - Don't forget to add a test function to the test class. + + - If a new test class is added, please declare it in the `PhoenixTestSuite` class. + +### Play in CLI + +```sql Review comment: @paul-rogers Thank you very much. Does it mean that the "sql" character can be removed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452211#comment-17452211 ] ASF GitHub Bot commented on DRILL-7863: --- paul-rogers commented on a change in pull request #2332: URL: https://github.com/apache/drill/pull/2332#discussion_r760821424 ## File path: contrib/storage-phoenix/README.md ## @@ -0,0 +1,98 @@ +# [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix + +## Description + + Phoenix say : "We put the SQL back in NoSQL", + Drill call : "We use the SQL to cross almost all the file systems and storage engines", + "Cheers !", users said. + +## Documentation + +Features : + + - Full support for Enhanced Vector Framework. + + - Tested in phoenix 4.14 and 5.1.2. + + - Support the array data type. + + - Support the pushdown (Project, Limit, Filter, Aggregate, Join, CrossJoin, Join_Filter, GroupBy, Distinct and more). + + - Use the PQS client (6.0). + +Related Information : + + 1. PHOENIX-6398: Returns uniform SQL dialect in calcite for the PQS + + 2. PHOENIX-6582: Bump default HBase version to 2.3.7 and 2.4.8 + + 3. PHOENIX-6605, PHOENIX-6606 and PHOENIX-6607. + + 4. DRILL-8060, DRILL-8061 and DRILL-8062. + + 5. [QueryServer 6.0.0-drill-r1](https://github.com/luocooong/phoenix-queryserver/releases/tag/6.0.0-drill-r1) + +## Testing + + The test framework of phoenix queryserver required the Hadoop 3, but exist `PHOENIX-5993` and `HBASE-22394` : + +``` +" The HBase PMC does not release multiple artifacts for both Hadoop2 and Hadoop3 support at the current time. +Current HBase2 releases still compile against Hadoop2 by default, and using Hadoop 3 against HBase2 +requires a recompilation of HBase because of incompatible changes between Hadoop2 and Hadoop3. " +``` + +### Recommended Practices + + 1. Download HBase 2.4.2 sources and rebuild with Hadoop 3. + + 2. Remove the `Ignore` annotation in `PhoenixTestSuite.java`. + + 3. Go to the phoenix root folder and run test. + +### To Add Features + + - Don't forget to add a test function to the test class. + + - If a new test class is added, please declare it in the `PhoenixTestSuite` class. + +### Play in CLI + +```sql Review comment: Nit: `text`. All the tables and what-not will throw off the SQL formatter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451824#comment-17451824 ] ASF GitHub Bot commented on DRILL-7863: --- dzamo commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-983644114 Dear PR author and reviewers. This is a generic message to say that we would like to merge this PR in time for the 1.20 release. Currently we're targeting a master branch freeze date of 2021-12-10 (10 Dec). Please strive to complete development and review by this time, or indicate that the PR will need more time (and how much). Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451124#comment-17451124 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-982640275 @paul-rogers @cgivre Thanks for your support. This plugin is getting better with your help. Could you please take a look again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450562#comment-17450562 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on a change in pull request #2332: URL: https://github.com/apache/drill/pull/2332#discussion_r758509422 ## File path: contrib/storage-phoenix/src/main/resources/logback-test.xml.bak ## @@ -0,0 +1,49 @@ + + + Review comment: The original plan was to delete it before merging the PR, but now I have deleted it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450555#comment-17450555
]
ASF GitHub Bot commented on DRILL-7863:
---
luocooong commented on a change in pull request #2332:
URL: https://github.com/apache/drill/pull/2332#discussion_r758505941
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixStoragePluginConfig.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.PlanStringBuilder;
+import org.apache.drill.common.logical.AbstractSecuredStoragePluginConfig;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.exec.store.security.CredentialProviderUtils;
+import org.apache.drill.exec.store.security.UsernamePasswordCredentials;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName(PhoenixStoragePluginConfig.NAME)
+public class PhoenixStoragePluginConfig extends
AbstractSecuredStoragePluginConfig {
+
+ public static final String NAME = "phoenix";
+ public static final String THIN_DRIVER_CLASS =
"org.apache.phoenix.queryserver.client.Driver";
+ public static final String FAT_DRIVER_CLASS =
"org.apache.phoenix.jdbc.PhoenixDriver";
+
+ private final String host;
+ private final int port;
+ private final String jdbcURL; // (options) Equal to host + port
+ private final Map props; // (options) See also
http://phoenix.apache.org/tuning.html
+
+ @JsonCreator
+ public PhoenixStoragePluginConfig(
+ @JsonProperty("host") String host,
+ @JsonProperty("port") int port,
+ @JsonProperty("username") String username,
Review comment:
As a side note, What is the difference between `opUserName` and
`queryUserName`?
```java
opUserName = scan.getUserName();
queryUserName = negotiator.context().getFragmentContext().getQueryUserName();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Storage Plugin for Apache Phoenix
> -
>
> Key: DRILL-7863
> URL: https://issues.apache.org/jira/browse/DRILL-7863
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Cong Luo
>Priority: Major
>
> There is a to-do list :
> # MVP on EVF.
> # Security Authentication.
> # Support both the thin(PQS) and fat(ZK) driver.
> # Compatibility with phoenix 4.x and 5.x.
> # Shaded dependencies.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450496#comment-17450496
]
ASF GitHub Bot commented on DRILL-7863:
---
cgivre commented on a change in pull request #2332:
URL: https://github.com/apache/drill/pull/2332#discussion_r758421339
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixStoragePluginConfig.java
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Objects;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.common.PlanStringBuilder;
+import org.apache.drill.common.logical.AbstractSecuredStoragePluginConfig;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.exec.store.security.CredentialProviderUtils;
+import org.apache.drill.exec.store.security.UsernamePasswordCredentials;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName(PhoenixStoragePluginConfig.NAME)
+public class PhoenixStoragePluginConfig extends
AbstractSecuredStoragePluginConfig {
+
+ public static final String NAME = "phoenix";
+ public static final String THIN_DRIVER_CLASS =
"org.apache.phoenix.queryserver.client.Driver";
+ public static final String FAT_DRIVER_CLASS =
"org.apache.phoenix.jdbc.PhoenixDriver";
+
+ private final String host;
+ private final int port;
+ private final String jdbcURL; // (options) Equal to host + port
+ private final Map props; // (options) See also
http://phoenix.apache.org/tuning.html
+
+ @JsonCreator
+ public PhoenixStoragePluginConfig(
+ @JsonProperty("host") String host,
+ @JsonProperty("port") int port,
+ @JsonProperty("username") String username,
+ @JsonProperty("password") String password,
+ @JsonProperty("jdbcURL") String jdbcURL,
+ @JsonProperty("credentialsProvider") CredentialsProvider
credentialsProvider,
+ @JsonProperty("props") Map props) {
+super(CredentialProviderUtils.getCredentialsProvider(username, password,
credentialsProvider), credentialsProvider == null);
+this.host = host;
+this.port = port == 0 ? 8765 : port;
+this.jdbcURL = jdbcURL;
+this.props = props == null ? Collections.emptyMap() : props;
+ }
+
+ @JsonIgnore
+ public UsernamePasswordCredentials getUsernamePasswordCredentials() {
+return new UsernamePasswordCredentials(credentialsProvider);
+ }
+
+ @JsonProperty("host")
+ public String getHost() {
+return host;
+ }
+
+ @JsonProperty("port")
+ public int getPort() {
+return port;
+ }
+
+ @JsonProperty("username")
+ public String getUsername() {
+if (directCredentials) {
+ return getUsernamePasswordCredentials().getUsername();
+}
+return null;
+ }
+
+ @JsonIgnore
+ @JsonProperty("password")
+ public String getPassword() {
+if (directCredentials) {
+ return getUsernamePasswordCredentials().getPassword();
+}
+return null;
+ }
+
+ @JsonProperty("jdbcURL")
+ public String getJdbcURL() {
+return jdbcURL;
+ }
+
+ @JsonProperty("props")
+ public Map getProps() {
+return props;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+if (o == this) {
+ return true;
+}
+if (o == null || !(o instanceof PhoenixStoragePluginConfig)) {
+ return false;
+}
+PhoenixStoragePluginConfig config = (PhoenixStoragePluginConfig) o;
+// URL first
+if (StringUtils.isNotBlank(config.getJdbcURL())) {
+ return Objects.equals(this.jdbcURL, config.getJdbcURL());
+}
+// Then the host and port
+return Objects.equals(this.host, config.getHost()) &&
Objects.equals(this.port, config.getPort());
+ }
+
+ @Override
+ public int hashCode() {
+if (StringUtils.isNotBlank(jdbcURL)) {
+ return Objects.hash(jdbcURL);
+}
+return Objects.hash(host, port);
+ }
+
+ @Override
+
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450247#comment-17450247
]
ASF GitHub Bot commented on DRILL-7863:
---
paul-rogers commented on a change in pull request #2332:
URL: https://github.com/apache/drill/pull/2332#discussion_r758097799
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixReader.java
##
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix;
+
+import java.math.BigDecimal;
+import java.sql.Array;
+import java.sql.Date;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.sql.Types;
+import java.util.Arrays;
+import java.util.Map;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.physical.resultSet.ResultSetLoader;
+import org.apache.drill.exec.physical.resultSet.RowSetLoader;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.vector.accessor.ColumnWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.shaded.guava.com.google.common.collect.Maps;
+
+public class PhoenixReader {
+
+ private final RowSetLoader writer;
+ private final ColumnDefn[] columns;
+ private final ResultSet results;
+ private long count;
+
+ public PhoenixReader(ResultSetLoader loader, ColumnDefn[] columns, ResultSet
results) {
+this.writer = loader.writer();
+this.columns = columns;
+this.results = results;
+ }
+
+ public RowSetLoader getStorage() {
+return writer;
+ }
+
+ public long getCount() {
+return count;
+ }
+
+ /**
+ * Fetch and process one row.
+ * @return return true if one row is processed, return false if there is no
next row.
+ * @throws SQLException
+ */
+ public boolean processRow() throws SQLException {
+if (results.next()) {
+ writer.start();
+ for (int index = 0; index < columns.length; index++) {
+if (columns[index].getSqlType() == Types.ARRAY) {
+ Array result = results.getArray(index + 1);
+ if (result != null) {
+columns[index].load(result.getArray());
+ }
+} else {
+ Object result = results.getObject(index + 1);
+ if (result != null) {
+columns[index].load(result);
+ }
+}
+ }
+ count++;
+ writer.save();
+ return true;
+}
+return false;
+ }
+
+ protected static final Map COLUMN_TYPE_MAP =
Maps.newHashMap();
+
+ static {
+// text
+COLUMN_TYPE_MAP.put(Types.VARCHAR, MinorType.VARCHAR);
+COLUMN_TYPE_MAP.put(Types.CHAR, MinorType.VARCHAR);
+// numbers
+COLUMN_TYPE_MAP.put(Types.BIGINT, MinorType.BIGINT);
+COLUMN_TYPE_MAP.put(Types.INTEGER, MinorType.INT);
+COLUMN_TYPE_MAP.put(Types.SMALLINT, MinorType.INT);
+COLUMN_TYPE_MAP.put(Types.TINYINT, MinorType.INT);
+COLUMN_TYPE_MAP.put(Types.DOUBLE, MinorType.FLOAT8);
+COLUMN_TYPE_MAP.put(Types.FLOAT, MinorType.FLOAT4);
+COLUMN_TYPE_MAP.put(Types.DECIMAL, MinorType.VARDECIMAL);
+// time
+COLUMN_TYPE_MAP.put(Types.DATE, MinorType.DATE);
+COLUMN_TYPE_MAP.put(Types.TIME, MinorType.TIME);
+COLUMN_TYPE_MAP.put(Types.TIMESTAMP, MinorType.TIMESTAMP);
+// binary
+COLUMN_TYPE_MAP.put(Types.BINARY, MinorType.VARBINARY); // Raw fixed
length byte array. Mapped to byte[].
+COLUMN_TYPE_MAP.put(Types.VARBINARY, MinorType.VARBINARY); // Raw variable
length byte array.
+// boolean
+COLUMN_TYPE_MAP.put(Types.BOOLEAN, MinorType.BIT);
+ }
+
+ protected abstract static class ColumnDefn {
+
+final String name;
+final int index;
+final int sqlType;
+ColumnWriter writer;
+
+public String getName() {
+ return name;
+}
+
+public int getIndex() {
+ return index;
+}
+
+public int getSqlType() {
+ return sqlType;
+}
+
+public ColumnDefn(String name, int index, int sqlType) {
+ this.name = name;
+ this.index = index;
+ this.sqlType = sqlType;
+}
+
+
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449309#comment-17449309
]
ASF GitHub Bot commented on DRILL-7863:
---
paul-rogers commented on a change in pull request #2332:
URL: https://github.com/apache/drill/pull/2332#discussion_r757120210
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixGroupScan.java
##
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.phoenix;
+
+import java.util.List;
+import java.util.Objects;
+
+import org.apache.drill.common.PlanStringBuilder;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.PhysicalOperatorSetupException;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.physical.base.ScanStats.GroupScanProperty;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+
+@JsonTypeName("phoenix-scan")
+public class PhoenixGroupScan extends AbstractGroupScan {
+
+ private final String sql;
+ private final List columns;
+ private final PhoenixScanSpec scanSpec;
+ private final double rows;
+ private final ScanStats scanStats;
+ private final PhoenixStoragePlugin plugin;
+
+ private int hashCode;
+
+ @JsonCreator
+ public PhoenixGroupScan(
+ @JsonProperty("sql") String sql,
+ @JsonProperty("columns") List columns,
+ @JsonProperty("scanSpec") PhoenixScanSpec scanSpec,
+ @JsonProperty("rows") double rows,
+ @JsonProperty("config") PhoenixStoragePluginConfig config,
+ @JacksonInject StoragePluginRegistry plugins) {
+super("no-user");
+this.sql = sql;
+this.columns = columns;
+this.scanSpec = scanSpec;
+this.rows = rows;
+this.scanStats = computeScanStats();
+this.plugin = plugins.resolve(config, PhoenixStoragePlugin.class);
+ }
+
+ public PhoenixGroupScan(PhoenixScanSpec scanSpec, PhoenixStoragePlugin
plugin) {
+super("no-user");
+this.sql = scanSpec.getSql();
+this.columns = ALL_COLUMNS;
+this.scanSpec = scanSpec;
+this.rows = 100;
Review comment:
The row count is used, I believe, to plan joins. Is 100 a good estimate?
A number this lows suggests to Drill that it can ship the results to all nodes
as part of a broadcast join. If the actual number of rows is 1M or 100M, that
will have turned out to be a poor choice.
No good solution here: you actually don't know the number of rows...
##
File path:
contrib/storage-phoenix/src/main/java/org/apache/drill/exec/store/phoenix/PhoenixScanBatchCreator.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ *
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438730#comment-17438730
]
ASF GitHub Bot commented on DRILL-7863:
---
Z0ltrix commented on pull request #2332:
URL: https://github.com/apache/drill/pull/2332#issuecomment-960959540
>
>
> > Hi @dzamo , phoenix uses keytab and other stuff within the connection
string: https://phoenix.apache.org/#connStr isnt this what we need?
> > Or, the other aspect... use code from PQS for the connection to
phoenix... they use some sort of proxyUser from
org.apache.hadoop.security.UserGroupInformation as well:
https://github.com/apache/phoenix-queryserver/blob/master/phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java#L553
> > Regards, Christian
>
> @Z0ltrix Okaaay. I'd forgotten that Phoenix lives on top of Hadoop anyway.
So if you've already enabled Hadoop's impersonation using the needed
`proxyuser` settings then maybe all that we need here is to tack a `doAs` onto
the JDBC URL you already have working (which might already include some
Kerberos bits).
>
> ```
> jdbc:phoenix:thin:url=http://localhost:8765?doAs=alice
> ```
>
> See https://phoenix.apache.org/server.html#Impersonation. @luocooong this
should be reasonably easy, let me know if I can help with anything. We should
imitate the behaviour of storage-hive which also contains "doAs". In my head
it's something like this farcically simplified pseudocode:
>
> ```
> if (config.getOption('drill.exec.impersonation.enabled') == true) {
> phoenixJdbcUrl = phoenixJdbcUrl + "?doAs=" + activeDrillUser;
> }
> ```
Hi @dzamo,
exaclty... as described in the original Issue #2296 we have the following
connection String for PQS
`jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF;authentication=SPNEGO;principal=drill/bit@MYCLUSTER;keytab=/etc/hadoop/conf/drill.keytab`
But this will not work with native phoenix connection, because there is no
doAs parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Storage Plugin for Apache Phoenix
> -
>
> Key: DRILL-7863
> URL: https://issues.apache.org/jira/browse/DRILL-7863
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Cong Luo
>Priority: Major
>
> There is a to-do list :
> # MVP on EVF.
> # Security Authentication.
> # Support both the thin(PQS) and fat(ZK) driver.
> # Compatibility with phoenix 4.x and 5.x.
> # Shaded dependencies.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[
https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438710#comment-17438710
]
ASF GitHub Bot commented on DRILL-7863:
---
dzamo edited a comment on pull request #2332:
URL: https://github.com/apache/drill/pull/2332#issuecomment-960890029
> Hi @dzamo , phoenix uses keytab and other stuff within the connection
string: https://phoenix.apache.org/#connStr isnt this what we need?
>
> Or, the other aspect... use code from PQS for the connection to phoenix...
they use some sort of proxyUser from
org.apache.hadoop.security.UserGroupInformation as well:
https://github.com/apache/phoenix-queryserver/blob/master/phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java#L553
>
> Regards, Christian
@Z0ltrix Okaaay. I'd forgotten that Phoenix lives on top of Hadoop anyway.
So if you've already enabled Hadoop's impersonation using the needed
`proxyuser` settings then maybe all that we need here is to tack a `doAs` onto
the JDBC URL you already have working (which might already include some
Kerberos bits).
```
jdbc:phoenix:thin:url=http://localhost:8765?doAs=alice
```
See https://phoenix.apache.org/server.html#Impersonation. @luocooong this
should be reasonably easy, let me know if I can help with anything. We should
imitate the behaviour of storage-hive which also contains "doAs". In my head
it's something like this farcically simplified pseudocode:
```
if (config.getOption('drill.exec.impersonation.enabled') == true) {
phoenixJdbcUrl = phoenixJdbcUrl + "?doAs=" + activeDrillUser;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add Storage Plugin for Apache Phoenix
> -
>
> Key: DRILL-7863
> URL: https://issues.apache.org/jira/browse/DRILL-7863
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Other
>Reporter: Cong Luo
>Assignee: Cong Luo
>Priority: Major
>
> There is a to-do list :
> # MVP on EVF.
> # Security Authentication.
> # Support both the thin(PQS) and fat(ZK) driver.
> # Compatibility with phoenix 4.x and 5.x.
> # Shaded dependencies.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438708#comment-17438708 ] ASF GitHub Bot commented on DRILL-7863: --- dzamo commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-960890029 > Hi @dzamo , phoenix uses keytab and other stuff within the connection string: https://phoenix.apache.org/#connStr isnt this what we need? > > Or, the other aspect... use code from PQS for the connection to phoenix... they use some sort of proxyUser from org.apache.hadoop.security.UserGroupInformation as well: https://github.com/apache/phoenix-queryserver/blob/master/phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java#L553 > > Regards, Christian @Z0ltrix Okaaay. I'd forgotten that Phoenix lives on top of Hadoop anyway. So if you've already enabled Hadoop's impersonation using the needed `proxyuser` settings then maybe all that we need here is to tack a `doAs` onto the JDBC URL you already have working (which might already include some Kerberos bits). ``` jdbc:phoenix:thin:url=http://localhost:8765?doAs=alice ``` See https://phoenix.apache.org/server.html#Impersonation. @luocooong this should be reasonably easy, let me know if I can help with anything... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438689#comment-17438689 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-960809785 > > > > Of course this PR is great even without working Impersonation, but for enterprise its necessary to let the enduser run the query against the storage. > > The fact that drill impersonates the proxyUser correctly against HDFS and HBase let me hope that this is also feasable for phoenix :) > > Hi @Z0ltrix. I believe that Drill uses the native support for impersonation that is built into Hadoop when it impersonates to HDFS, Hive, HBase (and MapR-DB through whatever is built into that). No other storage plugins support any impersonation yet. Also there is no support for propagating a Kerberos authn context (a TGT I guess?) over the hop from Drill to the external data source. But... we want it to be there! We are in a planning phase and talking to security guys and we want to start coding on this in November. Your input is very valuable so we'd like to keep checking in with you as we go. > > cc @cgivre Hi @dzamo , phoenix uses keytab and other stuff within the connection string: https://phoenix.apache.org/#connStr isnt this what we need? Or, the other aspect... use code from PQS for the connection to phoenix... they use some sort of proxyUser from org.apache.hadoop.security.UserGroupInformation as well: https://github.com/apache/phoenix-queryserver/blob/master/phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java#L553 Regards, Christian -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438685#comment-17438685 ] ASF GitHub Bot commented on DRILL-7863: --- dzamo commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-960802212 > > > > @dzamo I was under the impression that the vault work we did earlier actually did enable impersonation for most other storage plugins. The key component was the creation of the `credentialProvider` object which allowed that. Maybe @vvysotskyi could weigh in here as well. @cgivre @vvysotskyi Would be great to hear from him but my understanding is that what we achieved there was to remove the hardcoded credentials of what is still a shared service account from the storage config JSON. We didn't go so far as to impersonate the active Drill user. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438682#comment-17438682 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-960795904 > > Of course this PR is great even without working Impersonation, but for enterprise its necessary to let the enduser run the query against the storage. > > The fact that drill impersonates the proxyUser correctly against HDFS and HBase let me hope that this is also feasable for phoenix :) > > Hi @Z0ltrix. I believe that Drill uses the native support for impersonation that is built into Hadoop when it impersonates to HDFS, Hive, HBase (and MapR-DB through whatever is built into that). No other storage plugins support any impersonation yet. Also there is no support for propagating a Kerberos authn context (a TGT I guess?) over the hop from Drill to the external data source. But... we want it to be there! We are in a planning phase and talking to security guys and we want to start coding on this in November. Your input is very valuable so we'd like to keep checking in with you as we go. > > cc @cgivre @dzamo I was under the impression that the vault work we did earlier actually did enable impersonation for most other storage plugins. The key component was the creation of the `credentialProvider` object which allowed that. Maybe @vvysotskyi could weigh in here as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438677#comment-17438677 ] ASF GitHub Bot commented on DRILL-7863: --- dzamo commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-960776463 > Of course this PR is great even without working Impersonation, but for enterprise its necessary to let the enduser run the query against the storage. > > The fact that drill impersonates the proxyUser correctly against HDFS and HBase let me hope that this is also feasable for phoenix :) Hi @Z0ltrix. I believe that Drill uses the native support for impersonation that is built into Hadoop when it impersonates to HDFS, Hive, HBase (and MapR-DB through whatever is built into that). No other storage plugins support any impersonation yet. Also there is no support for propagating a Kerberos authn context (a TGT I guess?) over the hop from Drill to the external data source. But... we want it to be there! We are in a planning phase and talking to security guys and we want to start coding on this in November. Your input is very valuable so we'd like to keep checking in with you as we go. cc @cgivre -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438448#comment-17438448 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 > @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. Hi @luocooong , as you can see at the project downloads page, phoenix is shipped in several version-combinations. For the newest phoenix version > 5 its only build against hbase 2.x and hadoop 3.x https://github.com/apache/phoenix/blob/master/pom.xml#L78 In fact that hbase 2.3.7 is the current stable release and it recommends hadoop 2.10.0 or 3.1.2, and Drill is already on hadoop 3, i would go with that constellation. https://github.com/apache/hbase/blob/rel/2.3.7/pom.xml#L1349 phoenix == 5.1.2 PQS == 6.0.0 hbase == 2.3.7 hadoop >= 3.1.2 Hope this thoughts help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438445#comment-17438445 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix edited a comment on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 > @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. Hi @luocooong , as you can see at the project downloads page, phoenix is shipped in several version-combinations. For the newest phoenix version > 5 its only build against hbase 2.x and hadoop 3.x https://github.com/apache/phoenix/blob/master/pom.xml#L78 In fact that hbase 2.3.7 is the current stable release and it recommends hadoop 2.10.0 or 3.1.2, and Drill is already on hadoop 3, i would go with that constellation. https://github.com/apache/hbase/blob/rel/2.3.7/pom.xml#L1349 phoenix == 5.1.2 PQS == 6.0.0 hbase == 2.3.7 hadoop >= 3.1.2 Hope these thoughts help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438429#comment-17438429 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-958973388 @luocooong Thanks for this PR. This will be a really great addition to Drill! I wanted to suggest that since there isn't a testcontainer for Apache Phoenix, that it might be a better approach to use the Phoenix test classes. Here's one that I found: https://github.com/apache/phoenix/blob/master/phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java Maybe take a look and see what you think. From my recollection, this is what we had to do with the Splunk connector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438426#comment-17438426 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959421590 @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438399#comment-17438399 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-958973388 @luocooong Thanks for this PR. This will be a really great addition to Drill! I wanted to suggest that since there isn't a testcontainer for Apache Phoenix, that it might be a better approach to use the Phoenix test classes. Here's one that I found: https://github.com/apache/phoenix/blob/master/phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java Maybe take a look and see what you think. From my recollection, this is what we had to do with the Splunk connector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438394#comment-17438394 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438387#comment-17438387 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959421590 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438385#comment-17438385 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix edited a comment on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438366#comment-17438366 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-958973388 @luocooong Thanks for this PR. This will be a really great addition to Drill! I wanted to suggest that since there isn't a testcontainer for Apache Phoenix, that it might be a better approach to use the Phoenix test classes. Here's one that I found: https://github.com/apache/phoenix/blob/master/phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java Maybe take a look and see what you think. From my recollection, this is what we had to do with the Splunk connector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438212#comment-17438212 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix edited a comment on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 > @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. Hi @luocooong , as you can see at the project downloads page, phoenix is shipped in several version-combinations. For the newest phoenix version > 5 its only build against hbase 2.x and hadoop 3.x https://github.com/apache/phoenix/blob/master/pom.xml#L78 In fact that hbase 2.3.7 is the current stable release and it recommends hadoop 2.10.0 or 3.1.2, and Drill is already on hadoop 3, i would go with that constellation. https://github.com/apache/hbase/blob/rel/2.3.7/pom.xml#L1349 phoenix == 5.1.2 PQS == 6.0.0 hbase == 2.3.7 hadoop >= 3.1.2 Hope these thoughts help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438207#comment-17438207 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959753259 > @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. Hi @luocooong , as you can see at the project downloads page, phoenix is shipped in several version-combinations. For the newest phoenix version > 5 its only build against hbase 2.x and hadoop 3.x https://github.com/apache/phoenix/blob/master/pom.xml#L78 In fact that hbase 2.3.7 is the current stable release and it recommends hadoop 2.10.0 or 3.1.2, and Drill is already on hadoop 3, i would go with that constellation. https://github.com/apache/hbase/blob/rel/2.3.7/pom.xml#L1349 phoenix == 5.1.2 PQS == 6.0.0 hbase == 2.3.7 hadoop >= 3.1.2 Hope this thoughts help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438134#comment-17438134 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-959421590 @cgivre Thanks for the information. Actually, I have already tried to use the PQS test framework into Drill. Unfortunately, the Hadoop test dependencies (2.7) and HBase test dependencies (1.4) caused a lot of conflict (Drill use Hadoop 3.2 and HBase 2.4). That's one of the reasons I chose to make a container image. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438010#comment-17438010 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-958973388 @luocooong Thanks for this PR. This will be a really great addition to Drill! I wanted to suggest that since there isn't a testcontainer for Apache Phoenix, that it might be a better approach to use the Phoenix test classes. Here's one that I found: https://github.com/apache/phoenix/blob/master/phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java Maybe take a look and see what you think. From my recollection, this is what we had to do with the Splunk connector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429638#comment-17429638 ] ASF GitHub Bot commented on DRILL-7863: --- lgtm-com[bot] commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-944902469 This pull request **introduces 1 alert** when merging 3eeb5024df1f2e597a6e023a73235d8874785cb0 into 0c9451e6720e5028e1187067cc6d1957ff998bef - [view on LGTM.com](https://lgtm.com/projects/g/apache/drill/rev/pr-2656c660810b502c2182dde88b74accb84c1b550) **new alerts:** * 1 for Potential database resource leak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428855#comment-17428855 ] ASF GitHub Bot commented on DRILL-7863: --- cgivre commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-943446349 > @Z0ltrix Thanks for the suggestion. In the #2296 , you came up with one good idea, because I might also need it. The key point is that, how can we pass the user name dynamically, on-demand ? Since the last reply, I found a little useful docs : [Submitting Queries from the REST API with Impersonation](https://drill.apache.org/docs/submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled/) . > All right, that's progress. Do you mind if I complete this (pull request) to-do list before work on the impersonation feature ? @luocooong I'd actually suggest you take a look at the docs for the credential provider (https://github.com/apache/drill/blob/master/docs/dev/PluginCredentialsProvider.md#developer-notes). It's actually really straightforward to add. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428844#comment-17428844 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-943435016 @Z0ltrix Thanks for the suggestion. In the #2296 , you came up with one good idea, because I might also need it. The key point is that, how can we pass the user name dynamically, on-demand ? Since the last reply, I found a little useful docs : [Submitting Queries from the REST API with Impersonation](https://drill.apache.org/docs/submitting-queries-from-the-rest-api-when-impersonation-is-enabled-and-authentication-is-disabled/) . All right, that's progress. Do you mind if I complete this (pull request) to-do list before work on the impersonation feature ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428691#comment-17428691 ] ASF GitHub Bot commented on DRILL-7863: --- Z0ltrix commented on pull request #2332: URL: https://github.com/apache/drill/pull/2332#issuecomment-943135149 Hi @luocooong, great work! Looking forward to use this storage plugin. Have you thought about impersonation for this? https://phoenix.apache.org/server.html#Impersonation explains it for pqs with doAs parameter. For fat client i´m not sure... Maybe it has to be done manually, like in the pqs code: https://github.com/apache/phoenix-queryserver/blob/master/phoenix-queryserver/src/main/java/org/apache/phoenix/queryserver/server/QueryServer.java#L553 Do you think its possible to add this? Regards, Christian -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428418#comment-17428418 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong opened a new pull request #2332: URL: https://github.com/apache/drill/pull/2332 # [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix ## Description To-do lists : - [x] core module - [ ] testcontainers based on self-developed - [ ] bugfix and integration test in real world - [ ] code cleanup and javadocs ## Documentation WIP ## Testing WIP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7863) Add Storage Plugin for Apache Phoenix
[ https://issues.apache.org/jira/browse/DRILL-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428183#comment-17428183 ] ASF GitHub Bot commented on DRILL-7863: --- luocooong opened a new pull request #2332: URL: https://github.com/apache/drill/pull/2332 # [DRILL-7863](https://issues.apache.org/jira/browse/DRILL-7863): Add Storage Plugin for Apache Phoenix ## Description To-do lists : - [x] core module - [ ] testcontainers based on self-developed - [ ] bugfix and integration test in real world - [ ] code cleanup and javadocs ## Documentation WIP ## Testing WIP -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add Storage Plugin for Apache Phoenix > - > > Key: DRILL-7863 > URL: https://issues.apache.org/jira/browse/DRILL-7863 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > There is a to-do list : > # MVP on EVF. > # Security Authentication. > # Support both the thin(PQS) and fat(ZK) driver. > # Compatibility with phoenix 4.x and 5.x. > # Shaded dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
