[GitHub] drill issue #1007: MD-2888: Selecting a non-existing field from a MapR-DB JS...
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/1007 +1 ---
[GitHub] drill pull request #875: DRILL-5671 Set secure ACLs (Access Control List) fo...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127295205
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZKSecureACLProvider.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.coord.zk;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.curator.framework.api.ACLProvider;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import java.util.List;
+
+/**
+ * ZKSecureACLProvider restricts access to znodes created by Drill
+ * The cluster discovery znode i.e. the znode containing the list of
Drillbits is
+ * readable by anyone.
+ * For all other znodes only the creator of the znode, i.e the Drillbit
user, has full access.
+ */
+
+public class ZKSecureACLProvider implements ACLProvider {
+
+static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ZKSecureACLProvider.class);
+// Creator has full access
+static ImmutableList DEFAULT_ACL = new
ImmutableList.Builder()
+
.addAll(Ids.CREATOR_ALL_ACL.iterator())
+ .build();
+// Creator has full access
+// Everyone else has only read access
+static ImmutableList DRILL_CLUSTER_ACL = new
ImmutableList.Builder()
--- End diff --
Please use java-doc style comment block and mention why and where this ACL
is used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #875: DRILL-5671 Set secure ACLs (Access Control List) fo...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/875#discussion_r127295586
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -112,7 +112,8 @@ drill.exec: {
retry: {
count: 7200,
delay: 500
-}
+},
+use.secure_acl: false
--- End diff --
This needs to be set to true in
`/opt/mapr/drill/drill-1.11.0/conf/drill-distrib.conf` by `configure.sh` when
security is enabled on a MapR cluster. Do we have a tracking work item for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #700: DRILL-5137 - Optimize count(*) queries on MapR-DB B...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/700#discussion_r93487747
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
---
@@ -124,6 +124,11 @@ public HBaseRecordReader(Connection connection,
HBaseSubScan.HBaseSubScanSpec su
} else {
rowKeyOnly = false;
transformed.add(ROW_KEY_PATH);
+ /* DRILL-5137 - optimize count(*) queries on MapR-DB Binary tables */
+ if (isSkipQuery()) {
--- End diff --
Further optimization can be name by returning only a `count` vector in the
`next()` call, similar to
[this](https://github.com/apache/drill/blob/master/contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java#L203-L204).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #700: DRILL-5137 - Optimize count(*) queries on MapR-DB B...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/700#discussion_r93486944
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java
---
@@ -124,6 +124,11 @@ public HBaseRecordReader(Connection connection,
HBaseSubScan.HBaseSubScanSpec su
} else {
rowKeyOnly = false;
transformed.add(ROW_KEY_PATH);
+ /* DRILL-5137 - optimize count(*) queries on MapR-DB Binary tables */
--- End diff --
This branches into the else part of
`if (!isStarQuery()) {`
Can you verify if a query can be both Star and Skip query at the same time
when count(*) has been requested?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #688: DRILL-5119 - Update MapR version to 5.2.0.40963-mapr
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/688 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #678: DRILL-5065 - Optimize count( * ) queries on MapR-DB JSON T...
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/678 +1 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #677: DRILL-5065 - Optimize count( * ) queries on MapR-DB...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/677#discussion_r90932924
--- Diff:
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBFormatPluginConfig.java
---
@@ -65,6 +68,15 @@ public void setAllTextMode(boolean mode) {
allTextMode = mode;
}
+ @JsonProperty("disableCountOptimization")
+ public void setReadDocumentForCount(boolean mode) {
--- End diff --
setter function does not match the parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #670: DRILL-5065 - Optimize count( * ) queries on MapR-DB JSON T...
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/670 Could you please add couple of unit test to verify the plan? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #587: DRILL-4894: Fix unit test failure in 'storage-hive/core' m...
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/587 I have verified that this does not alter the content of binary package. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #587: DRILL-4894: Fix unit test failure in 'storage-hive/...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/587 DRILL-4894: Fix unit test failure in 'storage-hive/core' module Exclude 'hadoop-mapreduce-client-core' and 'hadoop-auth' as transitive dependencies from 'hbase-server' You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4894 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/587.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #587 commit f3c26e34e3a72ef338c4dbca1a0204f342176972 Author: Aditya Kishore <a...@apache.org> Date: 2016-09-16T19:14:35Z DRILL-4894: Fix unit test failure in 'storage-hive/core' module Exclude 'hadoop-mapreduce-client-core' and 'hadoop-auth' as transitive dependencies from 'hbase-server' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #586: DRILL-4886: Merge maprdb format plugin source code
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/586 DRILL-4886: Merge maprdb format plugin source code These commits add MapR Format Plugin to Drill code base. The default build/test/packaging behavior for mapr-format-plugin module are 1. BUILD of mapr-format-plugin is ENABLED. 2. Unit tests of mapr-format-plugin module are DISABLED (use `-Pmapr` to enable). 3. Packaging of mapr-format-plugin is DISABLED (use `-Pmapr` to enable). Please see [LEGAL-251](https://issues.apache.org/jira/browse/LEGAL-251) for discussion/conclusion regarding inclusion of source code with non-open-source dependency. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill maprdb-plugin-merge Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/586.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #586 commit cb6cc78b9991e9d6e384a2cc28b70332a9d92a22 Author: Patrick Wong <pw...@maprtech.com> Date: 2014-09-09T20:35:54Z Initial commit commit 95face3575817719a389fe049d3d8c1d65bb12dd Author: Steven Phillips <sphill...@maprtech.com> Date: 2014-09-23T18:45:27Z First version of maprdb plugin + Moved files under contrib/format-maprdb commit bfd27ec82851446a918b65f06805c63f4f0690ae Author: Patrick Wong <pw...@maprtech.com> Date: 2014-10-08T15:20:11Z Updated plugin version to 0.7.0-SNAPSHOT commit 846451926a9f9c43b874ed0cdfb565186419bef9 Author: Anbang Xu <a...@maprtech.com> Date: 2014-10-21T22:49:30Z MD-12: Filters are not being pushed down through maprdb plugin + Reverted mapr-hbase back to MapR 4.0.1 release + Apply DRILL-1508 and DRILL-1567 to MapRDB plugin 1. https://github.com/apache/drill/commit/67df8cfa27041c9696255ee96e2d066e034f6f58 2. https://github.com/apache/drill/commit/71f5ad447823cba96cf73fb6709f8ed6cc0b63e9 commit 0e5b67e964946380fc33f278f0a20340ec338109 Author: Patrick Wong <pw...@maprtech.com> Date: 2014-12-03T20:03:29Z Updated plugin version to 0.8.0-SNAPSHOT commit 7e59a1ee16bbff98f636dc290eda6bb8785c0ab2 Author: Aditya Kishore <adi...@mapr.com> Date: 2014-12-31T19:28:59Z DRILL-1900: Fix numeric overflow problem in HBase stats calculation. commit cb0a676dd542eaadba49776d22bdaf7b5b335720 Author: Aditya Kishore <adi...@mapr.com> Date: 2015-01-28T10:21:26Z MD-125: Update HBase storage plugin to support HBase 0.98 commit bed705556be5bd0ed55366620f116d93aaf2237e Author: vkorukanti <venki.koruka...@gmail.com> Date: 2015-02-06T15:18:06Z MD-143: Update DrillFileSystem package location after DRILL-2080. commit d5de0b5566370e84abbc9c8bbadb64b7e153 Author: Aditya Kishore <adi...@mapr.com> Date: 2015-02-25T01:12:08Z MD-131: Queries fail when using maprdb plugin commit b64ee82837d4ccb5d33d509787507bf9704e0820 Author: Aditya Kishore <adi...@mapr.com> Date: 2015-03-25T21:57:28Z MD-192: CONVERT_FROM in where clause * Set the writeIndex of ByteBuf returned by Unpooled.wrappedBuffer() to 0. commit 1c3013af5053136b167dd948f87f88092d2aa9d9 Author: Patrick Wong <pw...@maprtech.com> Date: 2015-04-15T20:01:32Z Updated plugin version to 0.9.0-SNAPSHOT commit 419e7f38f78428605000fd12873f4aa6eb7211c7 Author: vkorukanti <venki.koruka...@gmail.com> Date: 2015-04-22T00:51:37Z MD-244: Update MapRDB plugin after DRILL-2514 and DRILL-2413 commit c71ec6760ace959146174489e9551666880deb69 Author: Aditya Kishore <adi...@mapr.com> Date: 2015-04-22T18:02:15Z MD-245: Optiq is now Calcite. commit ec901a5f1412e9e119f243745b816532365d54fb Author: Patrick Wong <pw...@maprtech.com> Date: 2015-05-04T19:08:12Z Updated version to 1.0.0-SNAPSHOT commit 1c43e3d2f23cf96027d0b2940c7a6362880a5781 Author: Aditya Kishore <adi...@mapr.com> Date: 2015-05-06T00:28:37Z DRILL-2826: Simplify and centralize Operator cleanup commit d7fe7825ce833c37a320ab7b94b07f5440a5c69f Author: Aditya Kishore <adi...@mapr.com> Date: 2015-05-11T22:40:33Z MD-272: Incorporate DRILL-2848 (option to disable decimal data type) commit a19d0164a17952d44ece1e98b0f6192ecfb5b2e8 Author: Patrick Wong <pw...@maprtech.com> Date: 2015-05-20T21:52:48Z Updated plugin version to 1.1.0-SNAPSHOT +Using MapR v4.1.0 dependencies commit c5c24792b83f71826aea61ec9870146de48cfe98 Author: Aditya Kishore <a...@apache.org> Date: 2016-08-26T18:37:00Z MD-290: Fix build failure due to upstream changes. + Includes MD-292 commit ba8a785c9c932f7e9e90a28f1b67042e50126597 Author: Patrick Wong <pw...@maprtech.com> Date: 2015-07-06T03:30:03Z Updated version to 1.2.0-SNAPSHOT commit df
[GitHub] drill issue #556: DRILL-4819 - update MapR version to 5.2.0
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/556 LGTM. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #528: DRILL-4727: [Addendum] Exclude netty from HBase Cli...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/528 DRILL-4727: [Addendum] Exclude netty from HBase Client's transitive d⦠â¦ependencies Excluded `netty-all` from the list of transitive dependencies pulled by `mapr-hbase` in `mapr` profile. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4727 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #528 commit bd1d9c24f574af14541e7d4abcd386624de92dcc Author: Aditya Kishore <a...@apache.org> Date: 2016-06-17T22:13:48Z DRILL-4727: [Addendum] Exclude netty from HBase Client's transitive dependencies Excluded `netty-all` from the list of transitive dependencies pulled by `mapr-hbase` in `mapr` profile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #525: DRILL-4727: Exclude netty from HBase Client's trans...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/525 DRILL-4727: Exclude netty from HBase Client's transitive dependencies Excluded `netty-all` from the list of transitive dependencies pulled by `hbase-client`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4727 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #525 commit 3512043873a5b2a26f48a4005496afedd8da03b4 Author: Aditya Kishore <a...@apache.org> Date: 2016-06-16T20:39:21Z DRILL-4727: Exclude netty from HBase Client's transitive dependencies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #443: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/443 The latest and hopefully the last version of the patch (c2d9959e07f47a09a4a11c250f84f4874b7e1db4), modified the GuavaPatcher in a way to make it non-catastrophic failure to not able to patch Guava. This allows the JDBC driver to not crash when starting Drill in embedded mode (mostly used in unit tests). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #:
Github user adityakishore commented on the pull request: https://github.com/apache/drill/commit/323849e48fed967d43108e5cf85ba150a50f073e#commitcomment-17847479 In contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePluginConfig.java: In contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePluginConfig.java on line 46: The intended location is static block in Drillbit class[1] which is the bootstrap class for drillbits. Will remove it from here. [1] https://github.com/apache/drill/commit/323849e48fed967d43108e5cf85ba150a50f073e#diff-42d3952d93bea3940d7f15fb7bc95ef1R58 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #443: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/443 The latest patch, 323849e48fed967d43108e5cf85ba150a50f073e, moves the GuavaPatcher class to java-exec module and run the GuavaPatcher at Drillbit start (before the Stopwatch class is loaded somewhere else). @jacques-n Could you please take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #506: DRILL-4607: Add a split function that allows to separate s...
Github user adityakishore commented on the issue: https://github.com/apache/drill/pull/506 +1 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443 Unfortunately we can't, please see https://github.com/apache/drill/pull/443#issuecomment-200580166 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443 Addressed Jacques' comments on the previous patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/443#discussion_r65254026
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java
---
@@ -21,34 +21,36 @@
import java.util.Set;
import org.apache.calcite.schema.SchemaPlus;
-
import org.apache.drill.common.JSONOptions;
import org.apache.drill.exec.ops.OptimizerRulesContext;
import org.apache.drill.exec.server.DrillbitContext;
import org.apache.drill.exec.store.AbstractStoragePlugin;
import org.apache.drill.exec.store.SchemaConfig;
import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.Connection;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.collect.ImmutableSet;
public class HBaseStoragePlugin extends AbstractStoragePlugin {
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HBaseStoragePlugin.class);
+ private static final HBaseConnectionManager hbaseConnectionManager =
HBaseConnectionManager.INSTANCE;
--- End diff --
The HBaseConnectionManager is static, and is essentially a connection pool
which maintains one HBase connection per storage plugin instance as you
recommend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/443#discussion_r65127177
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java
---
@@ -175,7 +179,7 @@ public boolean hasNext() {
done = true;
}
} catch (IOException e) {
- throw new DrillRuntimeException("Caught error while fetching
rows from for table:" + Bytes.toString(table.getTableName()), e);
+ throw new DrillRuntimeException("Caught error while fetching
rows from for table:" + hbaseTableName, e);
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-222385259 The latest patch introduces a separate class `HBaseConnectionManager` and moves the (HBase) connection life-cycle management responsibility from HBaseStoragePlugin to this class. Also addressed Jacques' other comment regarding testing for connection close/abort in `getConnection()` method. Added test which verifies that HBase connection close or even HBase cluster restart does not impact running queries against HBase and does not require Drill cluster restart. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4603: Refactor FileSystem plugin code to...
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/476#discussion_r59603934
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPluginImplementationProvider.java
---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.dfs;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.ClassPathFileSystem;
+import org.apache.drill.exec.store.LocalSyncableFileSystem;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.DEFAULT_WS_NAME;
+
+/**
+ * Provides needed component by the {@link FileSystemPlugin}. This can be
overridden to supply customized components
+ * (such as custom schema factory) to {@link FileSystemPlugin}.
+ */
+public class FileSystemPluginImplementationProvider {
+
+ protected final FileSystemConfig fsConfig;
+ protected final String fsPluginName;
+ protected final DrillbitContext dContext;
+
+ private FormatCreator formatCreator; /** Don't use this directly, use
{@link #getFormatCreator(Configuration)} */
+
+ /**
+ * Instantiate an object
+ * @param dContext {@link DrillbitContext} instance.
+ * @param fsPluginName Name of the FileSystemPlugin storage
+ * @param fsConfig FileSystemPlugin configuration
+ */
+ public FileSystemPluginImplementationProvider(
+ final DrillbitContext dContext,
+ final String fsPluginName,
+ final FileSystemConfig fsConfig) {
+this.dContext = dContext;
+this.fsConfig = fsConfig;
+this.fsPluginName = fsPluginName;
+ }
+
+ /**
+ * @return Return any properties needed for {@link Configuration} object
that are not set in
+ * {@link FileSystemPlugin}'s configuration.
+ */
+ public Map<String, String> getFsProps() {
+return ImmutableMap.of(
+"fs.classpath.impl", ClassPathFileSystem.class.getName(),
+"fs.drill-local.impl", LocalSyncableFileSystem.class.getName()
+);
+ }
+
+ /**
+ * @return Create and return {@link FormatCreator} based on the storage
plugin configuration.
+ */
+ public FormatCreator getFormatCreator(final Configuration fsConf) {
+if (formatCreator == null) {
--- End diff --
Is the `FormatCreator` instance tied to some settings in `fsConf`?
If yes, wouldn't a subsequent invocation of this function ignore the passed
`fsConf` with different values of such settings?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/443#discussion_r59120474
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java
---
@@ -19,36 +19,75 @@
import java.io.IOException;
import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
import org.apache.calcite.schema.SchemaPlus;
-
import org.apache.drill.common.JSONOptions;
+import org.apache.drill.common.exceptions.UserException;
import org.apache.drill.exec.ops.OptimizerRulesContext;
import org.apache.drill.exec.server.DrillbitContext;
import org.apache.drill.exec.store.AbstractStoragePlugin;
import org.apache.drill.exec.store.SchemaConfig;
import org.apache.drill.exec.store.StoragePluginOptimizerRule;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
import com.google.common.collect.ImmutableSet;
+import com.google.common.util.concurrent.UncheckedExecutionException;
public class HBaseStoragePlugin extends AbstractStoragePlugin {
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HBaseStoragePlugin.class);
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HBaseStoragePlugin.class);
private final DrillbitContext context;
- private final HBaseStoragePluginConfig engineConfig;
+ private final HBaseStoragePluginConfig storeConfig;
private final HBaseSchemaFactory schemaFactory;
@SuppressWarnings("unused")
private final String name;
- public HBaseStoragePlugin(HBaseStoragePluginConfig configuration,
DrillbitContext context, String name)
+ private final LoadingCache<Integer, Connection> connectionCache;
+
+ public HBaseStoragePlugin(HBaseStoragePluginConfig storeConfig,
DrillbitContext context, String name)
throws IOException {
this.context = context;
this.schemaFactory = new HBaseSchemaFactory(this, name);
-this.engineConfig = configuration;
+this.storeConfig = storeConfig;
this.name = name;
+
+this.connectionCache = CacheBuilder.newBuilder()
+.expireAfterAccess(24, TimeUnit.HOURS)
+.removalListener(new RemovalListener<Integer, Connection>() {
+ @Override
+ public void onRemoval(RemovalNotification<Integer, Connection>
notification) {
+try {
+ notification.getValue().close();
+ logger.debug("HBase connection '{}' closed.",
notification.getValue());
+} catch (Throwable t) {
+ logger.warn("Error while closing HBase connection.", t);
+}
+ }
+})
+.build(new CacheLoader<Integer, Connection>() {
+ @Override
+ public synchronized Connection load(Integer key /* ignored */)
throws Exception {
+if (connection == null
+|| connection.isAborted()
+|| connection.isClosed()) {
+ connection =
ConnectionFactory.createConnection(HBaseStoragePlugin.this.storeConfig.getHBaseConf());
+ logger.debug("HBase connection '{}' created.", connection);
+}
+return connection;
+ }
+ private Connection connection = null;
--- End diff --
> Why is this an instance field and not protected?
Did you mean static?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-207022155 @jacques-n I'll wait for your final review before I merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-203699506 The latest patch, 1684daa5a081e1ae7d6d66cb5ee5e2f88015fc53, takes care of HBase connection life-cycle management within the plugin's life-cycle. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/443#discussion_r57384412
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java
---
@@ -25,36 +25,40 @@
import java.util.Map.Entry;
import java.util.NoSuchElementException;
-import com.google.common.collect.Iterators;
import org.apache.drill.common.exceptions.DrillRuntimeException;
import org.apache.drill.exec.store.sys.BasePersistentStore;
import org.apache.drill.exec.store.sys.PersistentStoreConfig;
import org.apache.drill.exec.store.sys.PersistentStoreMode;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
+import com.google.common.collect.Iterators;
+
public class HBasePersistentStore extends BasePersistentStore {
--- End diff --
Okay, I'd like to do that as a separate JIRA once I send a feeler out on
the user list, just in case someone is using it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-200991092 I'll take a look at the plugin life-cycle model and see if we can tie HBase connection life-cycle to it. As far as I know HBase follows a per-user connection model so I'll have to test how does that play with Drill impersonation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on a diff in the pull request:
https://github.com/apache/drill/pull/443#discussion_r57376985
--- Diff:
contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/config/HBasePersistentStore.java
---
@@ -25,36 +25,40 @@
import java.util.Map.Entry;
import java.util.NoSuchElementException;
-import com.google.common.collect.Iterators;
import org.apache.drill.common.exceptions.DrillRuntimeException;
import org.apache.drill.exec.store.sys.BasePersistentStore;
import org.apache.drill.exec.store.sys.PersistentStoreConfig;
import org.apache.drill.exec.store.sys.PersistentStoreMode;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
+import com.google.common.collect.Iterators;
+
public class HBasePersistentStore extends BasePersistentStore {
--- End diff --
Do you mean getting rid of HBase persistent store altogether?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-4199: Add Support for HBase 1.X
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/443#issuecomment-200580166 `jcl-over-slf4j` works as long as the application use only the surface logging APIs. Unfortunately, HBase uses some [internal classes](https://github.com/apache/hbase/blob/branch-1.3/hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpRequestLog.java) from the `commons-logging` library which are not available in `jcl-over-slf4j`. I have verified that the dependency does not traverse to other module and certainly does not get bundled with the binary tarball. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4050: Add zip archives to the list of ar...
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/249#issuecomment-196542189 This enhanced version of the script allows integrated download and verification of a Drill release. It can be used to verify both the main release artifacts and maven repository artifacts. For example, to verify the 1.6 rc0 release artifacts, I ran ``` ./verify_release.sh https://repository.apache.org/content/repositories/orgapachedrill-1030/ /tmp/drill-1.6/maven/ ./verify_release.sh http://home.apache.org/~parthc/drill/releases/1.6.0/rc0/ /tmp/drill-1.6/main/ ``` If I had pre-downloaded the files in the respective folders, I'd run ``` ./verify_release.sh /tmp/drill-1.6/maven/ ./verify_release.sh /tmp/drill-1.6/main/ ``` Finally, run with `-nv` option to reduce the verbosity of the output. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4501: Complete MapOrListWriter for all s...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/427 DRILL-4501: Complete MapOrListWriter for all supported data types You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4501 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #427 commit f3e0c68d2562dc6e081c3943864dd8a110072144 Author: Aditya Kishore <a...@apache.org> Date: 2016-03-12T19:12:34Z DRILL-4501: Complete MapOrListWriter for all supported data types --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4493 - Fixed issues in various POMs with...
Github user adityakishore commented on a diff in the pull request: https://github.com/apache/drill/pull/421#discussion_r55577997 --- Diff: exec/jdbc/pom.xml --- @@ -134,6 +134,7 @@ org.apache.hadoop hadoop-common + test --- End diff -- hadoop-common is not a dependency for jdbc or jdbc-all project. [drill-jdbc-all-1.5.0.jar](https://repo.maven.apache.org/maven2/org/apache/drill/exec/drill-jdbc-all/1.5.0/drill-jdbc-all-1.5.0.jar) does not contain any Hadoop classes and if you look at other profiles (default-hadoop, cdh, hdp) in this pom, they all have this in test scope. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4493 - Fixed issues in various POMs with...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/421 DRILL-4493 - Fixed issues in various POMs with MapR profile You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4493 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #421 commit 2445d642c484f6216596be813033dc4d9799124b Author: Aditya Kishore <a...@apache.org> Date: 2016-03-09T08:38:33Z DRILL-4493 - Fixed issues in various POMs with MapR profile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4491: FormatPluginOptionsDescriptor requ...
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/418#issuecomment-194097775 Until I looked at the code, I was under assumption that we are using Jackson to extract the serializable properties. We can, and should, definitely go that route. The way code currently works is that it iterate through allthe table options and see if there is a Java field present in the corresponding FormatPluginConfig class. If it does find one, and this is why I say it is a bug in the current implementation, it makes is accessible (`setAccesible(true)`) implying that it is expected to work with non-public fields and sets the value to the one passed as parameter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4491: FormatPluginOptionsDescriptor requ...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/418 DRILL-4491: FormatPluginOptionsDescriptor requires FormatPluginConfig⦠⦠fields to be public You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4491 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/418.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #418 commit cce8467c2476da871891bad7db6cab3236537f7c Author: Aditya Kishore <a...@apache.org> Date: 2016-03-09T00:49:55Z DRILL-4491: FormatPluginOptionsDescriptor requires FormatPluginConfig fields to be public --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-4050: Add zip archives to the list of ar...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/249 DRILL-4050: Add zip archives to the list of artifacts in verify_relea⦠â¦se.sh You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-4050 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/249.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #249 commit dac1acb66bdc0c24794df0cacef834c7abdb3944 Author: Aditya Kishore <a...@apache.org> Date: 2015-11-08T22:08:15Z DRILL-4050: Add zip archives to the list of artifacts in verify_release.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-3888: Build test jars for all Drill Modu...
Github user adityakishore commented on the pull request: https://github.com/apache/drill/pull/188#issuecomment-146310855 Thanks for the review. Since we are most likely going to have another RC for 1.2, I am planning to merge it with current master so that the test artifacts get published for 1.2 release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request: DRILL-3888: Build test jars for all Drill Modu...
GitHub user adityakishore opened a pull request: https://github.com/apache/drill/pull/188 DRILL-3888: Build test jars for all Drill Modules This patch moves the test jar configuration to the root pom and remove it from individual module's pom. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adityakishore/drill DRILL-3888 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/188.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #188 commit 57f2bacfb0a7539ed2ec8ae3740a9e82e524141c Author: Aditya Kishore <a...@apache.org> Date: 2015-10-02T18:36:45Z DRILL-3888: Build test jars for all Drill Modules --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
