[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes
[ https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088350#comment-16088350 ] ASF GitHub Bot commented on DRILL-5601: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/860#discussion_r127537900 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java --- @@ -265,7 +277,7 @@ private StreamingAggregator createAggregatorInternal() throws SchemaChangeExcept context.getFunctionRegistry(), context.getOptions()); cg.getCodeGenerator().plainJavaCapable(true); // Uncomment out this line to debug the generated code. -//cg.getCodeGenerator().saveCodeForDebugging(true); +cg.getCodeGenerator().saveCodeForDebugging(true); --- End diff -- Comment out ... > Rollup of External Sort memory management fixes > --- > > Key: DRILL-5601 > URL: https://issues.apache.org/jira/browse/DRILL-5601 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.12.0 > > > Rollup of a set of specific JIRA entries that all relate to the very > difficult problem of managing memory within Drill in order for the external > sort to stay within a memory budget. In general, the fixes relate to better > estimating memory used by the three ways that Drill allocates vector memory > (see DRILL-5522) and to predicting the size of vectors that the sort will > create, to avoid repeated realloc-copy cycles (see DRILL-5594). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes
[
https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088348#comment-16088348
]
ASF GitHub Bot commented on DRILL-5601:
---
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/860#discussion_r127536602
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java
---
@@ -111,6 +120,8 @@ public void buildSchema() throws SchemaChangeException {
case STOP:
state = BatchState.STOP;
return;
+ default:
+break;
--- End diff --
Not getting into a "religious" debate, but "default+break" is just a
useless no-op
> Rollup of External Sort memory management fixes
> ---
>
> Key: DRILL-5601
> URL: https://issues.apache.org/jira/browse/DRILL-5601
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
>
> Rollup of a set of specific JIRA entries that all relate to the very
> difficult problem of managing memory within Drill in order for the external
> sort to stay within a memory budget. In general, the fixes relate to better
> estimating memory used by the three ways that Drill allocates vector memory
> (see DRILL-5522) and to predicting the size of vectors that the sort will
> create, to avoid repeated realloc-copy cycles (see DRILL-5594).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5659) C++ Client (master) behavior is unstable resulting incorrect result or exception in API calls
[ https://issues.apache.org/jira/browse/DRILL-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088258#comment-16088258 ] ASF GitHub Bot commented on DRILL-5659: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/876 Still haven't been able to confirm the SASL encryption case, but there is no reason why this fix would affect that. Merging it in. > C++ Client (master) behavior is unstable resulting incorrect result or > exception in API calls > - > > Key: DRILL-5659 > URL: https://issues.apache.org/jira/browse/DRILL-5659 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Rob Wu >Assignee: Parth Chandra > Labels: ready-to-commit > Fix For: 1.11.0 > > Attachments: 1-10cppClient-1.10.0Drillbit-hive.log, > 1-10cppClient-1.10.0Drillbit-metadata and catalog test.log, > 1-10cppClient-1.9.0Drillbit-dfs.log, 1-10cppClient-1.9.0Drillbit-hive.log, > 1-10cppClient-1.9.0Drillbit-metadata and catalog test.log, > 1-11cppClient-1.10.0Drillbit-hive.log, 1-11cppClient-1.10.0Drillbit-metadata > and catalog test.log, 1-11cppClient-1.9.0Drillbit-dfs.log, > 1-11cppClient-1.9.0Drillbit-hive.log, 1-11cppClient-1.9.0Drillbit-metadata > and catalog test.log > > > I recently compiled the C++ client (on windows) from master and tested > against a 1.9.0 drillbit. The client's behavior does not meet the stable > release requirement. > Some API functionalities are broken and should be investigated. > Most noticeable is the getColumns(...) call. It will throw an exception with > "Cannot decode getcolumns results" when the number of rows (column records) > exceeds a certain number. > I also noticed that: during query execution + data retrieval, if the table is > large enough, the result coming back will start to become NULL or empty. > I will see if I can generate some drillclient logs to put in the attachment. > I will also compile and test on other platforms. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (DRILL-5515) "IS NO DISTINCT FROM" and it's equivalent form aren't handled likewise
[
https://issues.apache.org/jira/browse/DRILL-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Muhammad Gelbana closed DRILL-5515.
---
Resolution: Invalid
What I posted is NOT the equivalent form for the "IS DISTINCT FROM" clause.
> "IS NO DISTINCT FROM" and it's equivalent form aren't handled likewise
> --
>
> Key: DRILL-5515
> URL: https://issues.apache.org/jira/browse/DRILL-5515
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Muhammad Gelbana
>
> The following query fails to execute
> {code:sql}SELECT * FROM (SELECT `UserID` FROM `dfs`.`path_ot_parquet` tc)
> `t0` INNER JOIN (SELECT `UserID` FROM `dfs`.`path_ot_parquet` tc) `t1` ON
> (`t0`.`UserID` IS NOT DISTINCT FROM `t1`.`UserID`){code}
> and produces the following error message
> {noformat}org.apache.drill.common.exceptions.UserRemoteException:
> UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due to
> either a cartesian join or an inequality join [Error Id:
> 0bd41e06-ccd7-45d6-a038-3359bf5a4a7f on mgelbana-incorta:31010]{noformat}
> While the query's equivalent form runs fine
> {code:sql}SELECT * FROM (SELECT `UserID` FROM `dfs`.`path_ot_parquet` tc)
> `t0` INNER JOIN (SELECT `UserID` FROM `dfs`.`path_ot_parquet` tc) `t1` ON
> (`t0`.`UserID` = `t1`.`UserID` OR (`t0`.`UserID` IS NULL AND `t1`.`UserID` IS
> NULL)){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087982#comment-16087982
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user bitblender commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127540793
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.framework.imps.DefaultACLProvider;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.ExecConstants;
+
+
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
+
+public static ACLProvider getACLProvider(DrillConfig config, String
clusterId, String zkRoot) {
--- End diff --
Will make that change.
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[ https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087979#comment-16087979 ] ASF GitHub Bot commented on DRILL-5671: --- Github user bitblender commented on a diff in the pull request: https://github.com/apache/drill/pull/875#discussion_r127540711 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java --- @@ -39,6 +39,7 @@ String ZK_TIMEOUT = "drill.exec.zk.timeout"; String ZK_ROOT = "drill.exec.zk.root"; String ZK_REFRESH = "drill.exec.zk.refresh"; + String ZK_SECURE_ACL = "drill.exec.zk.use.secure_acl"; --- End diff -- I was going to make this "zk.use_secure_acl" but when I was looking for an example I found "bit.use.ip", so I modeled it on that. I will change this to "zk.apply_secure_acl" > Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster > > > Key: DRILL-5671 > URL: https://issues.apache.org/jira/browse/DRILL-5671 > Project: Apache Drill > Issue Type: New Feature > Components: Server >Reporter: Karthikeyan Manivannan >Assignee: Karthikeyan Manivannan > > All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e. > anyone gets to do CDRWA(create, delete, read, write, admin access). This > means that even on a secure cluster anyone can perform all CRDWA actions on > the znodes. > This should be changed such that: > - In a non-secure cluster, Drill will continue using the current default > [world:all] ACL > - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the > authenticated user that created the znode gets full access. The discovery > znodes i.e. the znodes with the list of Drillbits will have an additional > [world:read] ACL, i.e. the list of Drillbits will be readable by anyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087847#comment-16087847
]
ASF GitHub Bot commented on DRILL-5663:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/874#discussion_r127529165
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java
---
@@ -264,19 +265,26 @@ private ServerConnector createHttpsConnector() throws
Exception {
final SslContextFactory sslContextFactory = new SslContextFactory();
-if (config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH) &&
-
!Strings.isNullOrEmpty(config.getString(ExecConstants.HTTP_KEYSTORE_PATH))) {
- logger.info("Using configured SSL settings for web server");
-
sslContextFactory.setKeyStorePath(config.getString(ExecConstants.HTTP_KEYSTORE_PATH));
-
sslContextFactory.setKeyStorePassword(config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD));
-
- // TrustStore and TrustStore password are optional
- if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PATH)) {
-
sslContextFactory.setTrustStorePath(config.getString(ExecConstants.HTTP_TRUSTSTORE_PATH));
-if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PASSWORD)) {
-
sslContextFactory.setTrustStorePassword(config.getString(ExecConstants.HTTP_TRUSTSTORE_PASSWORD));
-}
+final boolean hasPath =
config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH);
--- End diff --
To avoid this double-checking, standard practice is:
* Use a blank value to indicate the value is unset.
* Provide a default (blank) value in `drill-module.conf`
Then, the code can just be:
```
String path = config.getString(ExecConstants.HTTP_KEYSTORE_PATH).trim();
if (! path.isEmpty()) {
// do stuff
```
Also, if the user is expected to set this key, please provide an example in
`drill-override-example.conf`.
> Drillbit fails to start when only keystore path is provided without keystore
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
> Fix For: 1.11.0
>
>
> When we configure keystore path without keystore password inside
> drill-override.conf for WebServer, then Drillbit fails to start. We should
> explicitly check for either both being present or both being absent. If any
> one of them is only present then throw startup exception for Drill.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087849#comment-16087849
]
ASF GitHub Bot commented on DRILL-5663:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/874#discussion_r127530311
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java
---
@@ -289,30 +297,19 @@ private ServerConnector createHttpsConnector() throws
Exception {
final DateTime now = DateTime.now();
// Create builder for certificate attributes
- final X500NameBuilder nameBuilder =
- new X500NameBuilder(BCStyle.INSTANCE)
- .addRDN(BCStyle.OU, "Apache Drill (auth-generated)")
- .addRDN(BCStyle.O, "Apache Software Foundation
(auto-generated)")
- .addRDN(BCStyle.CN,
workManager.getContext().getEndpoint().getAddress());
+ final X500NameBuilder nameBuilder = new
X500NameBuilder(BCStyle.INSTANCE).addRDN(BCStyle.OU, "Apache Drill
(auth-generated)").addRDN(BCStyle.O, "Apache Software Foundation
(auto-generated)").addRDN(BCStyle.CN,
workManager.getContext().getEndpoint().getAddress());
--- End diff --
Can we keep the previous one-argument-per-line, indented style? Much easier
to read.
> Drillbit fails to start when only keystore path is provided without keystore
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
> Fix For: 1.11.0
>
>
> When we configure keystore path without keystore password inside
> drill-override.conf for WebServer, then Drillbit fails to start. We should
> explicitly check for either both being present or both being absent. If any
> one of them is only present then throw startup exception for Drill.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087846#comment-16087846
]
ASF GitHub Bot commented on DRILL-5663:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/874#discussion_r127529372
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java
---
@@ -264,19 +265,26 @@ private ServerConnector createHttpsConnector() throws
Exception {
final SslContextFactory sslContextFactory = new SslContextFactory();
-if (config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH) &&
-
!Strings.isNullOrEmpty(config.getString(ExecConstants.HTTP_KEYSTORE_PATH))) {
- logger.info("Using configured SSL settings for web server");
-
sslContextFactory.setKeyStorePath(config.getString(ExecConstants.HTTP_KEYSTORE_PATH));
-
sslContextFactory.setKeyStorePassword(config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD));
-
- // TrustStore and TrustStore password are optional
- if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PATH)) {
-
sslContextFactory.setTrustStorePath(config.getString(ExecConstants.HTTP_TRUSTSTORE_PATH));
-if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PASSWORD)) {
-
sslContextFactory.setTrustStorePassword(config.getString(ExecConstants.HTTP_TRUSTSTORE_PASSWORD));
-}
+final boolean hasPath =
config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH);
+final boolean hasPassword =
config.hasPath(ExecConstants.HTTP_KEYSTORE_PASSWORD);
+
+// Check if both keypath and password are present or not
+if (hasPath && hasPassword) {
+ final String pathValue =
config.getString(ExecConstants.HTTP_KEYSTORE_PATH);
+ final String passwordValue =
config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD);
+
+ // checking if any one of them is null or empty
+ if (!Strings.isNullOrEmpty(pathValue) &&
!Strings.isNullOrEmpty(passwordValue)) {
--- End diff --
Path can't be null here, so `! pathValue.isEmpty()` works also.
> Drillbit fails to start when only keystore path is provided without keystore
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
> Fix For: 1.11.0
>
>
> When we configure keystore path without keystore password inside
> drill-override.conf for WebServer, then Drillbit fails to start. We should
> explicitly check for either both being present or both being absent. If any
> one of them is only present then throw startup exception for Drill.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5663) Drillbit fails to start when only keystore path is provided without keystore password.
[
https://issues.apache.org/jira/browse/DRILL-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087848#comment-16087848
]
ASF GitHub Bot commented on DRILL-5663:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/874#discussion_r127530151
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/WebServer.java
---
@@ -264,19 +265,26 @@ private ServerConnector createHttpsConnector() throws
Exception {
final SslContextFactory sslContextFactory = new SslContextFactory();
-if (config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH) &&
-
!Strings.isNullOrEmpty(config.getString(ExecConstants.HTTP_KEYSTORE_PATH))) {
- logger.info("Using configured SSL settings for web server");
-
sslContextFactory.setKeyStorePath(config.getString(ExecConstants.HTTP_KEYSTORE_PATH));
-
sslContextFactory.setKeyStorePassword(config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD));
-
- // TrustStore and TrustStore password are optional
- if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PATH)) {
-
sslContextFactory.setTrustStorePath(config.getString(ExecConstants.HTTP_TRUSTSTORE_PATH));
-if (config.hasPath(ExecConstants.HTTP_TRUSTSTORE_PASSWORD)) {
-
sslContextFactory.setTrustStorePassword(config.getString(ExecConstants.HTTP_TRUSTSTORE_PASSWORD));
-}
+final boolean hasPath =
config.hasPath(ExecConstants.HTTP_KEYSTORE_PATH);
+final boolean hasPassword =
config.hasPath(ExecConstants.HTTP_KEYSTORE_PASSWORD);
+
+// Check if both keypath and password are present or not
+if (hasPath && hasPassword) {
+ final String pathValue =
config.getString(ExecConstants.HTTP_KEYSTORE_PATH);
+ final String passwordValue =
config.getString(ExecConstants.HTTP_KEYSTORE_PASSWORD);
+
+ // checking if any one of them is null or empty
+ if (!Strings.isNullOrEmpty(pathValue) &&
!Strings.isNullOrEmpty(passwordValue)) {
+sslContextFactory.setKeyStorePath(pathValue);
+sslContextFactory.setKeyStorePassword(passwordValue);
+ }
+
+ // Throwing an exception if anyone of them is null or empty
+ else {
+throw new DrillbitStartupException("keystorepath and/or
keystorepassword can't be empty.");
--- End diff --
Please provide a bit of a clearer message. Also, provide the actual path
name. This will be the critical message for folks that made mistake, the
Drillbit won't start, and they have to figure out what's what.
Maybe,
```
"To enable web UI security, both " + ExecConstants.HTTP_KEYSTORE_PATH + "
and " + ... + " are required.";
```
Also, this is just for the Web UI? In that case, maybe just disable the web
UI, but don't fail the Drillbit.
```
"Web UI misconfiguration: web UI is disabled."
```
> Drillbit fails to start when only keystore path is provided without keystore
> password.
> --
>
> Key: DRILL-5663
> URL: https://issues.apache.org/jira/browse/DRILL-5663
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Sorabh Hamirwasia
>Assignee: Sindhuri Ramanarayan Rayavaram
> Fix For: 1.11.0
>
>
> When we configure keystore path without keystore password inside
> drill-override.conf for WebServer, then Drillbit fails to start. We should
> explicitly check for either both being present or both being absent. If any
> one of them is only present then throw startup exception for Drill.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087818#comment-16087818
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127525673
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.framework.imps.DefaultACLProvider;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.ExecConstants;
+
+
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
+
+public static ACLProvider getACLProvider(DrillConfig config, String
clusterId, String zkRoot) {
--- End diff --
See comment below: probably want to pass a root path here rather than the
components.
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087819#comment-16087819
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127528429
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a
secure installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of
Drillbits is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit
user, has full access.
+ *
+ */
+
+public class ZKSecureACLProvider implements ACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else
only read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String clusterName;
+final String drillZkRoot;
+final String drillClusterPath;
+
+public ZKSecureACLProvider(String clusterName, String drillZKRoot) {
+this.clusterName = clusterName;
+this.drillZkRoot = drillZKRoot;
+this.drillClusterPath = "/" + this.drillZkRoot + "/" +
this.clusterName ;
+}
+
+public List getDefaultAcl() {
+return DEFAULT_ACL;
+}
+
+public List getAclForPath(String path) {
--- End diff --
This approach encodes the meaning of paths in just the path name. Seems
fragile.
Alternatives:
* Register the secure (or insecure) paths so that the ZK cluster
coordinator (not this class) decides on security.
* Get ACL base on type: `getSecureACL()` or `getPublicACL()`, and let the
cluster coordinator define which is which.
* As a refinement of the first idea, provide some kind of config option to
externalize the set of paths and their security.
For example, in Drill-on-YARN, the app master creates ZK entries to ensure
that only one app master starts per cluster. We would not want to encode that
path information here.
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access).
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087822#comment-16087822
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127525830
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKACLProviderFactory.java
---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.curator.framework.imps.DefaultACLProvider;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.exec.ExecConstants;
+
+
+public class ZKACLProviderFactory {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKACLProviderFactory.class);
+
+public static ACLProvider getACLProvider(DrillConfig config, String
clusterId, String zkRoot) {
+if (config.getBoolean(ExecConstants.ZK_SECURE_ACL)) {
+if
(config.getBoolean(ExecConstants.USER_AUTHENTICATION_ENABLED)){
+logger.trace("ZKACLProviderFactory: Using secure ZK ACL");
--- End diff --
The logger inserts the class name for you; no need to repeat it here.
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087820#comment-16087820
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127524590
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -39,6 +39,7 @@
String ZK_TIMEOUT = "drill.exec.zk.timeout";
String ZK_ROOT = "drill.exec.zk.root";
String ZK_REFRESH = "drill.exec.zk.refresh";
+ String ZK_SECURE_ACL = "drill.exec.zk.use.secure_acl";
--- End diff --
`use.secure_acl` --> `secure_acl` as `use` is not a group. This provides:
```
zk: {
connect: "localhost:2181",
root: "drill",
secure_acl: true,
```
Are we using ACL's for security? Then, we are really "applying" ACLs, so
maybe `apply_acl` (apply an access control list...)
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5671) Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
[
https://issues.apache.org/jira/browse/DRILL-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087821#comment-16087821
]
ASF GitHub Bot commented on DRILL-5671:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127526692
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill in a
secure installation.
+ *
+ * The cluster discovery znode i.e. the znode containing the list of
Drillbits is
+ * readable by anyone.
+ *
+ * For all other znodes, only the creator of the znode, i.e the Drillbit
user, has full access.
+ *
+ */
+
+public class ZKSecureACLProvider implements ACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+
+/**
+ * DEFAULT_ACL gives the creator of a znode full-access to it
+ */
+static ImmutableList DEFAULT_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+/**
+ * DRILL_CLUSTER_ACL gives the creator full access and everyone else
only read access.
+ * Used on the Drillbit discovery znode (znode path
//)
+ * i.e. the node that contains the list of Drillbits in the cluster.
+ */
+ static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.READ_ACL_UNSAFE.iterator())
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+.build();
+final String clusterName;
+final String drillZkRoot;
+final String drillClusterPath;
+
+public ZKSecureACLProvider(String clusterName, String drillZKRoot) {
+this.clusterName = clusterName;
+this.drillZkRoot = drillZKRoot;
+this.drillClusterPath = "/" + this.drillZkRoot + "/" +
this.clusterName ;
--- End diff --
Encoding path building here seems fragile. The ZK cluster coordinator
should build the path, then pass that along to the factory, which passes it
here. Simplifies things if we ever have to change how we construct the path if
we create it in one place rather than two (or more).
> Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster
>
>
> Key: DRILL-5671
> URL: https://issues.apache.org/jira/browse/DRILL-5671
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
>Reporter: Karthikeyan Manivannan
>Assignee: Karthikeyan Manivannan
>
> All Drill ZK nodes, currently, are assigned a default [world:all] ACL i.e.
> anyone gets to do CDRWA(create, delete, read, write, admin access). This
> means that even on a secure cluster anyone can perform all CRDWA actions on
> the znodes.
> This should be changed such that:
> - In a non-secure cluster, Drill will continue using the current default
> [world:all] ACL
> - In a secure cluster, all nodes should have an [authid: all] ACL i.e. the
> authenticated user that created the znode gets full access. The discovery
> znodes i.e. the znodes with the list of Drillbits will have an additional
> [world:read] ACL, i.e. the list of Drillbits will be readable by anyone.
[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes
[ https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087816#comment-16087816 ] ASF GitHub Bot commented on DRILL-5601: --- Github user Ben-Zvi commented on a diff in the pull request: https://github.com/apache/drill/pull/860#discussion_r127363488 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/RecordBatchSizer.java --- @@ -70,52 +72,90 @@ */ public final int estSize; + +/** + * Number of times the value here (possibly repeated) appears in + * the record batch. + */ + public final int valueCount; -public final int entryCount; -public final int dataSize; -public final int estElementCount; + +/** + * Number of times the entries of this column appears. If this is a + * scalar, the entry count is the same as the value count. If this --- End diff -- If a scalar, entryCount would be 1 . > Rollup of External Sort memory management fixes > --- > > Key: DRILL-5601 > URL: https://issues.apache.org/jira/browse/DRILL-5601 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.12.0 > > > Rollup of a set of specific JIRA entries that all relate to the very > difficult problem of managing memory within Drill in order for the external > sort to stay within a memory budget. In general, the fixes relate to better > estimating memory used by the three ways that Drill allocates vector memory > (see DRILL-5522) and to predicting the size of vectors that the sort will > create, to avoid repeated realloc-copy cycles (see DRILL-5594). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4264) Dots in identifier are not escaped correctly
[
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087705#comment-16087705
]
Paul Rogers commented on DRILL-4264:
Thanks for the explanation! Can you suggest a solution (from the user's view)
that offers the best trade-off between a number of requirements?
Requirements:
* Allow dots in any name (schema, workspace, table, column, directory, file).
* Consistent behavior for all names.
* Compatible with existing queries.
* Familiar to users of similar systems (Hive, Impala, etc.)
Maybe propose a syntax and user-visible rules that achieve the above (as best
we can.) Once we agree on those rules, we can move on to discuss how we'll
implement the rules given Calcite and the Drill execution classes.
> Dots in identifier are not escaped correctly
>
>
> Key: DRILL-4264
> URL: https://issues.apache.org/jira/browse/DRILL-4264
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Reporter: Alex
>Assignee: Volodymyr Vysotskyi
>
> If you have some json data like this...
> {code:javascript}
> {
> "0.0.1":{
> "version":"0.0.1",
> "date_created":"2014-03-15"
> },
> "0.1.2":{
> "version":"0.1.2",
> "date_created":"2014-05-21"
> }
> }
> {code}
> ... there is no way to select any of the rows since their identifiers contain
> dots and when trying to select them, Drill throws the following error:
> Error: SYSTEM ERROR: UnsupportedOperationException: Unhandled field reference
> "0.0.1"; a field reference identifier must not have the form of a qualified
> name
> This must be fixed since there are many json data files containing dots in
> some of the keys (e.g. when specifying version numbers etc)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5672) Move session option setting from cluster fixture to client fixture
Paul Rogers created DRILL-5672: -- Summary: Move session option setting from cluster fixture to client fixture Key: DRILL-5672 URL: https://issues.apache.org/jira/browse/DRILL-5672 Project: Apache Drill Issue Type: Improvement Affects Versions: 1.10.0 Reporter: Paul Rogers Assignee: Paul Rogers Priority: Minor Fix For: 1.12.0 The recently-added "cluster fixture" allows setting session and system options. At present, both are set on the cluster fixture. However, it makes more sense for session options to be set on the client. This way, we can create two clients, each with different session options, for tests that need that flexibility. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4264) Dots in identifier are not escaped correctly
[
https://issues.apache.org/jira/browse/DRILL-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087229#comment-16087229
]
Volodymyr Vysotskyi commented on DRILL-4264:
On the one hand user knows that {{tmp}} workspace inside {{dfs}} plugin, so
user expects that query
{code:sql}
use dfs.tmp;
{code}
should work. On the other hand query
{code:sql}
show schemas;
{code}
returns the schema name {{dfs.tmp}}. So user also expects that the query with
such schema name should work.
{code:sql}
use `dfs.tmp`;
{code}
Both these cases work since schema name and its path are the same.
Schemas names with dots currently does not work in Drill. It is due to the
handling schema paths in [this
way|https://github.com/apache/drill/blob/3e8b01d5b0d3013e3811913f0fd6028b22c1ac3f/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserSession.java#L201]
Queries
{code:sql}
SELECT `dfs`.`ds`.`foo.json`.`a`.`b.c` FROM `dfs`.`ds`.`foo.json` (1)
SELECT `dfs.ds.foo.json.a`.`b.c` FROM `dfs.ds`.`foo.json` (2)
SELECT `dfs.ds.foo.json.a.b.c` FROM `dfs.ds.foo.json` (3)
{code}
will not work since Drill allows only table names or aliases before the field
names.
Considering only from clause, third case would not work, since Drill assumes
that {{`dfs.ds.foo.json`}} is the schema name only.
Queries on the directories with dots and table names with dots also works
correctly.
Hive has an
[option|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.support.quoted.identifiers]
that allows dots in the columns names (by default dots in the columns is
allowed).
Parquet also allows field names with dots.
Also current version of Drill can create parquet files with dots in field
names, but Drill will fail when querying this file.
> Dots in identifier are not escaped correctly
>
>
> Key: DRILL-4264
> URL: https://issues.apache.org/jira/browse/DRILL-4264
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Codegen
>Reporter: Alex
>Assignee: Volodymyr Vysotskyi
>
> If you have some json data like this...
> {code:javascript}
> {
> "0.0.1":{
> "version":"0.0.1",
> "date_created":"2014-03-15"
> },
> "0.1.2":{
> "version":"0.1.2",
> "date_created":"2014-05-21"
> }
> }
> {code}
> ... there is no way to select any of the rows since their identifiers contain
> dots and when trying to select them, Drill throws the following error:
> Error: SYSTEM ERROR: UnsupportedOperationException: Unhandled field reference
> "0.0.1"; a field reference identifier must not have the form of a qualified
> name
> This must be fixed since there are many json data files containing dots in
> some of the keys (e.g. when specifying version numbers etc)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5634) Add Crypto and Hash Functions
[ https://issues.apache.org/jira/browse/DRILL-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5634: Labels: doc-impacting ready-to-commit (was: doc-impacting) > Add Crypto and Hash Functions > - > > Key: DRILL-5634 > URL: https://issues.apache.org/jira/browse/DRILL-5634 > Project: Apache Drill > Issue Type: New Feature > Components: Functions - Drill >Affects Versions: 1.10.0 >Reporter: Charles Givre > Labels: doc-impacting, ready-to-commit > Fix For: 1.11.0 > > > This library contains a collection of cryptography-related functions for > Apache Drill. It generally mirrors the crypto functions in MySQL. The > package includes: > * **`aes_encrypt()`/ `aes_decrypt()`**: implement encryption and decryption > of data using the official AES (Advanced Encryption Standard) algorithm, > previously known as “Rijndael.” > `AES_ENCRYPT()` encrypts the string `str` using the key string `key_str` and > returns a binary string containing the encrypted output. `AES_DECRYPT()` > decrypts the encrypted string `crypt_str` using the key string `key_str` and > returns the original cleartext string. If either function argument is NULL, > the function returns NULL. > ```sql > > SELECT aes_encrypt( 'encrypted_text', 'my_secret_key' ) AS aes FROM > > (VALUES(1)); > +---+ > |aes| > +---+ > | JkcBUNAn8ByKWCcVmNrKMA== | > +---+ > > SELECT aes_encrypt( 'encrypted_text', 'my_secret_key' ) AS encrypted, > aes_decrypt(aes_encrypt( 'encrypted_text', 'my_secret_key' > ),'my_secret_key') AS decrypted > FROM (VALUES(1)); > +---+-+ > | encrypted |decrypted| > +---+-+ > | JkcBUNAn8ByKWCcVmNrKMA== | encrypted_text | > +---+-+ > ``` > * **`md5()`**: Returns the md5 hash of the text. > (https://en.wikipedia.org/wiki/MD5) > Usage: > ```sql > > select md5( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | ae2b1fca515949e5d54fb22b8ed95575 | > +---+ > ``` > * **`sha(`) / `sha1()`**: Calculates an SHA-1 160-bit checksum > for the string, as described in RFC 3174 (Secure Hash Algorithm). > (https://en.wikipedia.org/wiki/SHA-1) The value is returned as a string of > 40 hexadecimal digits, or NULL if the argument was NULL. Note that `sha()` > and `sha1()` are aliases for the same function. > ```sql > > select sha1( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | dc724af18fbdd4e59189f5fe768a5f8311527050 | > +---+ > ``` > * **`sha2(`) / `sha256()`**: Calculates an SHA-2 256-bit checksum > for the string. (https://en.wikipedia.org/wiki/SHA-2) The value is returned > as a string of hexadecimal digits, or NULL if the argument was NULL. Note > that `sha2()` and `sha256()` are aliases for the same function. > ```sql > > select sha2( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | cf80cd8aed482d5d1527d7dc72fceff84e6326592848447d2dc0b0e87dfc9a90 | > +---+ > ``` > Additionally, there are also `sha384()` and `sha512()` functions > which return SHA-2 hashes with 384 and 512 bit checksums. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5634) Add Crypto and Hash Functions
[ https://issues.apache.org/jira/browse/DRILL-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087150#comment-16087150 ] ASF GitHub Bot commented on DRILL-5634: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/865 +1, checked the latest changes and ran unit tests. > Add Crypto and Hash Functions > - > > Key: DRILL-5634 > URL: https://issues.apache.org/jira/browse/DRILL-5634 > Project: Apache Drill > Issue Type: New Feature > Components: Functions - Drill >Affects Versions: 1.10.0 >Reporter: Charles Givre > Labels: doc-impacting > Fix For: 1.11.0 > > > This library contains a collection of cryptography-related functions for > Apache Drill. It generally mirrors the crypto functions in MySQL. The > package includes: > * **`aes_encrypt()`/ `aes_decrypt()`**: implement encryption and decryption > of data using the official AES (Advanced Encryption Standard) algorithm, > previously known as “Rijndael.” > `AES_ENCRYPT()` encrypts the string `str` using the key string `key_str` and > returns a binary string containing the encrypted output. `AES_DECRYPT()` > decrypts the encrypted string `crypt_str` using the key string `key_str` and > returns the original cleartext string. If either function argument is NULL, > the function returns NULL. > ```sql > > SELECT aes_encrypt( 'encrypted_text', 'my_secret_key' ) AS aes FROM > > (VALUES(1)); > +---+ > |aes| > +---+ > | JkcBUNAn8ByKWCcVmNrKMA== | > +---+ > > SELECT aes_encrypt( 'encrypted_text', 'my_secret_key' ) AS encrypted, > aes_decrypt(aes_encrypt( 'encrypted_text', 'my_secret_key' > ),'my_secret_key') AS decrypted > FROM (VALUES(1)); > +---+-+ > | encrypted |decrypted| > +---+-+ > | JkcBUNAn8ByKWCcVmNrKMA== | encrypted_text | > +---+-+ > ``` > * **`md5()`**: Returns the md5 hash of the text. > (https://en.wikipedia.org/wiki/MD5) > Usage: > ```sql > > select md5( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | ae2b1fca515949e5d54fb22b8ed95575 | > +---+ > ``` > * **`sha(`) / `sha1()`**: Calculates an SHA-1 160-bit checksum > for the string, as described in RFC 3174 (Secure Hash Algorithm). > (https://en.wikipedia.org/wiki/SHA-1) The value is returned as a string of > 40 hexadecimal digits, or NULL if the argument was NULL. Note that `sha()` > and `sha1()` are aliases for the same function. > ```sql > > select sha1( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | dc724af18fbdd4e59189f5fe768a5f8311527050 | > +---+ > ``` > * **`sha2(`) / `sha256()`**: Calculates an SHA-2 256-bit checksum > for the string. (https://en.wikipedia.org/wiki/SHA-2) The value is returned > as a string of hexadecimal digits, or NULL if the argument was NULL. Note > that `sha2()` and `sha256()` are aliases for the same function. > ```sql > > select sha2( 'testing' ) from (VALUES(1)); > +---+ > | EXPR$0 | > +---+ > | cf80cd8aed482d5d1527d7dc72fceff84e6326592848447d2dc0b0e87dfc9a90 | > +---+ > ``` > Additionally, there are also `sha384()` and `sha512()` functions > which return SHA-2 hashes with 384 and 512 bit checksums. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5659) C++ Client (master) behavior is unstable resulting incorrect result or exception in API calls
[ https://issues.apache.org/jira/browse/DRILL-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5659: Labels: ready-to-commit (was: ) > C++ Client (master) behavior is unstable resulting incorrect result or > exception in API calls > - > > Key: DRILL-5659 > URL: https://issues.apache.org/jira/browse/DRILL-5659 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Rob Wu >Assignee: Parth Chandra > Labels: ready-to-commit > Fix For: 1.11.0 > > Attachments: 1-10cppClient-1.10.0Drillbit-hive.log, > 1-10cppClient-1.10.0Drillbit-metadata and catalog test.log, > 1-10cppClient-1.9.0Drillbit-dfs.log, 1-10cppClient-1.9.0Drillbit-hive.log, > 1-10cppClient-1.9.0Drillbit-metadata and catalog test.log, > 1-11cppClient-1.10.0Drillbit-hive.log, 1-11cppClient-1.10.0Drillbit-metadata > and catalog test.log, 1-11cppClient-1.9.0Drillbit-dfs.log, > 1-11cppClient-1.9.0Drillbit-hive.log, 1-11cppClient-1.9.0Drillbit-metadata > and catalog test.log > > > I recently compiled the C++ client (on windows) from master and tested > against a 1.9.0 drillbit. The client's behavior does not meet the stable > release requirement. > Some API functionalities are broken and should be investigated. > Most noticeable is the getColumns(...) call. It will throw an exception with > "Cannot decode getcolumns results" when the number of rows (column records) > exceeds a certain number. > I also noticed that: during query execution + data retrieval, if the table is > large enough, the result coming back will start to become NULL or empty. > I will see if I can generate some drillclient logs to put in the attachment. > I will also compile and test on other platforms. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5659) C++ Client (master) behavior is unstable resulting incorrect result or exception in API calls
[ https://issues.apache.org/jira/browse/DRILL-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5659: Affects Version/s: 1.11.0 > C++ Client (master) behavior is unstable resulting incorrect result or > exception in API calls > - > > Key: DRILL-5659 > URL: https://issues.apache.org/jira/browse/DRILL-5659 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.11.0 >Reporter: Rob Wu >Assignee: Parth Chandra > Labels: ready-to-commit > Fix For: 1.11.0 > > Attachments: 1-10cppClient-1.10.0Drillbit-hive.log, > 1-10cppClient-1.10.0Drillbit-metadata and catalog test.log, > 1-10cppClient-1.9.0Drillbit-dfs.log, 1-10cppClient-1.9.0Drillbit-hive.log, > 1-10cppClient-1.9.0Drillbit-metadata and catalog test.log, > 1-11cppClient-1.10.0Drillbit-hive.log, 1-11cppClient-1.10.0Drillbit-metadata > and catalog test.log, 1-11cppClient-1.9.0Drillbit-dfs.log, > 1-11cppClient-1.9.0Drillbit-hive.log, 1-11cppClient-1.9.0Drillbit-metadata > and catalog test.log > > > I recently compiled the C++ client (on windows) from master and tested > against a 1.9.0 drillbit. The client's behavior does not meet the stable > release requirement. > Some API functionalities are broken and should be investigated. > Most noticeable is the getColumns(...) call. It will throw an exception with > "Cannot decode getcolumns results" when the number of rows (column records) > exceeds a certain number. > I also noticed that: during query execution + data retrieval, if the table is > large enough, the result coming back will start to become NULL or empty. > I will see if I can generate some drillclient logs to put in the attachment. > I will also compile and test on other platforms. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
