[jira] [Commented] (DRILL-4959) Drill 1.8 not able to connect to S3
[ https://issues.apache.org/jira/browse/DRILL-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607484#comment-15607484 ] Gopal Nagar commented on DRILL-4959: Hi Uwe L. Korn, Could you please provide some update on this issue ? Thanks & Regards, Gopal Nagar > Drill 1.8 not able to connect to S3 > --- > > Key: DRILL-4959 > URL: https://issues.apache.org/jira/browse/DRILL-4959 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Gopal Nagar > > Hi Team, > I have followed below details to integrate Drill with AWS S3. Query keep > running for hours and doesn't display any output (I am querying only 2 row > file from S3). > Reference link > --- > https://abhishek-tiwari.com/post/reflections-on-apache-drill > https://drill.apache.org/docs/s3-storage-plugin/ > Query Format (Tried from UI & CLI) > > select * from `s3`.`hive.csv` LIMIT 10; > select * from `s3`.`bucket_name/hive.csv` LIMIT 10; > After seeing below log, I tried including jets3t-0.9.3.jar in jars directory > but it doesn't fix my problem. > Log Details > -- > 2016-10-24 17:00:02,461 [27f1c1ec-d82e-ba2a-2840-e7104320418f:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27f1c1ec-d82e-ba2a-2840-e7104320418f: select * from `s3`.`hive.csv` LIMIT 10 > 2016-10-24 17:00:02,479 [drill-executor-39] ERROR > o.a.d.exec.server.BootStrapContext - > org.apache.drill.exec.work.foreman.Foreman.run() leaked an exception. > java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zelaine Fong reassigned DRILL-4964: --- Assignee: Jinfeng Ni (was: Sorabh Hamirwasia) Assigning to [~jni] for review. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Jinfeng Ni > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.ops.QueryContext.getRo
[jira] [Commented] (DRILL-4674) Allow casting to boolean the same literals as in Postgre
[
https://issues.apache.org/jira/browse/DRILL-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607105#comment-15607105
]
ASF GitHub Bot commented on DRILL-4674:
---
Github user gparai commented on a diff in the pull request:
https://github.com/apache/drill/pull/610#discussion_r85036858
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/BooleanType.java
---
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals is formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html";>Postgre
Documentation}
+ */
+public enum BooleanType {
--- End diff --
Is this the desired package to declare BooleanType.java?
> Allow casting to boolean the same literals as in Postgre
>
>
> Key: DRILL-4674
> URL: https://issues.apache.org/jira/browse/DRILL-4674
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Data Types
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Drill does not return results when we try to cast 0 and 1 to boolean inside a
> value constructor.
> Drill version : 1.7.0-SNAPSHOT commit ID : 09b26277
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> values(cast(1 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 1
> Fragment 0:0
> [Error Id: 35dcc4bb-0c5d-466f-8fb5-cf7f0a892155 on centos-02.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> values(cast(0 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Where as we get results on Postgres for same query.
> {noformat}
> postgres=# values(cast(1 as boolean));
> column1
> -
> t
> (1 row)
> postgres=# values(cast(0 as boolean));
> column1
> -
> f
> (1 row)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-05-13 07:16:16,578 [28ca80bf-0af9-bc05-258b-6b5744739ed8:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException:
> Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.la
[jira] [Commented] (DRILL-4674) Allow casting to boolean the same literals as in Postgre
[
https://issues.apache.org/jira/browse/DRILL-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607108#comment-15607108
]
ASF GitHub Bot commented on DRILL-4674:
---
Github user gparai commented on a diff in the pull request:
https://github.com/apache/drill/pull/610#discussion_r85036792
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/BooleanType.java
---
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals is formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html";>Postgre
Documentation}
+ */
+public enum BooleanType {
+ TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+ FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+ private int numericValue;
+ private List literals;
+
+ BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+ }
+
+ public int getNumericValue() {
+ return numericValue;
+}
+
+ public List getLiterals() {
+ return literals;
+}
+
+ /**
+ * Finds boolean by passed literal.
+ * Leading or trailing whitespace is ignored, and case does not matter.
+ *
+ * @param literal boolean string representation
+ * @return boolean type
+ * @throws IllegalArgumentException if boolean type is not found
+ */
+ public static BooleanType get(String literal) {
+String value = literal.toLowerCase().trim();
+for (BooleanType booleanType : BooleanType.values()) {
+ if(booleanType.getLiterals().contains(value)) {
--- End diff --
if (
> Allow casting to boolean the same literals as in Postgre
>
>
> Key: DRILL-4674
> URL: https://issues.apache.org/jira/browse/DRILL-4674
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Data Types
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Drill does not return results when we try to cast 0 and 1 to boolean inside a
> value constructor.
> Drill version : 1.7.0-SNAPSHOT commit ID : 09b26277
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> values(cast(1 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 1
> Fragment 0:0
> [Error Id: 35dcc4bb-0c5d-466f-8fb5-cf7f0a892155 on centos-02.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> values(cast(0 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Where as we get results on Postgres for same query.
> {noformat}
> postgres=# values(cast(1 as boolean));
> column1
> -
> t
> (1 row)
> postgres=# values(cast(0 as boolean));
> column1
> -
> f
> (1 row)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-05-13 07:16:16,578 [28ca80bf-0af9-bc05-258b-6b5744739ed8:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException:
> Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> at
[jira] [Commented] (DRILL-4674) Allow casting to boolean the same literals as in Postgre
[
https://issues.apache.org/jira/browse/DRILL-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607106#comment-15607106
]
ASF GitHub Bot commented on DRILL-4674:
---
Github user gparai commented on a diff in the pull request:
https://github.com/apache/drill/pull/610#discussion_r85038037
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/sql/TestSimpleCastFunctions.java
---
@@ -18,18 +18,108 @@
package org.apache.drill.exec.sql;
import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.exceptions.UserRemoteException;
import org.junit.Test;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertThat;
+
public class TestSimpleCastFunctions extends BaseTestQuery {
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(TestSimpleCastFunctions.class);
@Test
- public void castFromBoolean() throws Exception {
-test("select cast(false as varchar(5)), cast(true as varchar(4)),
cast((1 < 5) as varchar(4)) from sys.options limit 1;");
+ public void testCastFromBooleanToString() throws Exception {
+testBuilder()
+.sqlQuery("select" +
+" cast(false as varchar(5)) c1," +
+" cast(true as varchar(4)) c2," +
+" cast((1 < 5) as varchar(4)) c3," +
+" cast((1 > 5) as varchar(5)) c4" +
+" from (values(1))")
+.unOrdered()
+.baselineColumns("c1", "c2", "c3", "c4")
+.baselineValues("false", "true", "true", "false")
+.go();
+ }
+
+ @Test
+ public void testCastStringToBooleanTrueValue() throws Exception {
+List literals = Arrays.asList("t", "true", "y", "yes", "on",
"1");
--- End diff --
Would be good to also check for case insensitivity here and in
testCastStringToBooleanFalseValue?
> Allow casting to boolean the same literals as in Postgre
>
>
> Key: DRILL-4674
> URL: https://issues.apache.org/jira/browse/DRILL-4674
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Data Types
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Drill does not return results when we try to cast 0 and 1 to boolean inside a
> value constructor.
> Drill version : 1.7.0-SNAPSHOT commit ID : 09b26277
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> values(cast(1 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 1
> Fragment 0:0
> [Error Id: 35dcc4bb-0c5d-466f-8fb5-cf7f0a892155 on centos-02.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> values(cast(0 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Where as we get results on Postgres for same query.
> {noformat}
> postgres=# values(cast(1 as boolean));
> column1
> -
> t
> (1 row)
> postgres=# values(cast(0 as boolean));
> column1
> -
> f
> (1 row)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-05-13 07:16:16,578 [28ca80bf-0af9-bc05-258b-6b5744739ed8:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException:
> Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
[jira] [Commented] (DRILL-4674) Allow casting to boolean the same literals as in Postgre
[
https://issues.apache.org/jira/browse/DRILL-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607107#comment-15607107
]
ASF GitHub Bot commented on DRILL-4674:
---
Github user gparai commented on a diff in the pull request:
https://github.com/apache/drill/pull/610#discussion_r85036776
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/BooleanType.java
---
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.impl;
+
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Enum that contains two boolean types: TRUE and FALSE.
+ * Each has numeric representation and list of allowed literals.
+ * List of literals is formed according to
+ * {@link https://www.postgresql.org/docs/9.6/static/datatype-boolean.html";>Postgre
Documentation}
+ */
+public enum BooleanType {
+ TRUE(1, Arrays.asList("t", "true", "y", "yes", "on", "1")),
+ FALSE(0, Arrays.asList("f", "false", "n", "no", "off", "0"));
+
+ private int numericValue;
+ private List literals;
+
+ BooleanType(int numericValue, List literals) {
+this.numericValue = numericValue;
+this.literals = literals;
+ }
+
+ public int getNumericValue() {
+ return numericValue;
--- End diff --
Please correct alignment here and in getLiterals()
> Allow casting to boolean the same literals as in Postgre
>
>
> Key: DRILL-4674
> URL: https://issues.apache.org/jira/browse/DRILL-4674
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Data Types
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
> Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Drill does not return results when we try to cast 0 and 1 to boolean inside a
> value constructor.
> Drill version : 1.7.0-SNAPSHOT commit ID : 09b26277
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> values(cast(1 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 1
> Fragment 0:0
> [Error Id: 35dcc4bb-0c5d-466f-8fb5-cf7f0a892155 on centos-02.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp> values(cast(0 as boolean));
> Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Where as we get results on Postgres for same query.
> {noformat}
> postgres=# values(cast(1 as boolean));
> column1
> -
> t
> (1 row)
> postgres=# values(cast(0 as boolean));
> column1
> -
> f
> (1 row)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-05-13 07:16:16,578 [28ca80bf-0af9-bc05-258b-6b5744739ed8:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException:
> Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException: Invalid value for boolean: 0
> Fragment 0:0
> [Error Id: 2dbcafe2-92c7-475e-a2aa-9745ef72c1cc on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
> [drill-java-exec-1.7.0-
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607092#comment-15607092 ] ASF GitHub Bot commented on DRILL-4964: --- GitHub user sohami reopened a pull request: https://github.com/apache/drill/pull/628 DRILL-4964: Drill fails to connect to hive metastore after hive metas… …tore is restarted unless drillbits are restarted also Changes: For certain methods in DrillHiveMetaStoreClient like getAllDatabases and getAllTables the HiveMetaStoreClient wraps the exception of type MetaException and TException both into MetaException. So in case of connection failure which is thrown as TException it is difficult to categorize at DrillClient level. Hence the fix is to close older connection and reconnect in case of these 2 api's. In all other cases proper set of exceptions are thrown where we can handle each one individually. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sohami/drill DRILL-4964 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #628 commit 95477151e9a5b28a121ffe6a879006e4333039c4 Author: Sorabh Hamirwasia Date: 2016-10-26T01:52:01Z DRILL-4964: Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also Changes: For certain methods in DrillHiveMetaStoreClient like getAllDatabases and getAllTables the HiveMetaStoreClient wraps the exception of type MetaException and TException both into MetaException. So in case of connection failure which is thrown as TException it is difficult to categorize at DrillClient level. Hence the fix is to close older connection and reconnect in case of these 2 api's. In all other cases proper set of exceptions are thrown where we can handle each one individually. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607091#comment-15607091 ] ASF GitHub Bot commented on DRILL-4964: --- Github user sohami closed the pull request at: https://github.com/apache/drill/pull/628 > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607090#comment-15607090 ] ASF GitHub Bot commented on DRILL-4964: --- Github user sohami commented on the issue: https://github.com/apache/drill/pull/628 @jinfengni - Can you please help to review ? > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[guava-18.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[guava-18.0.jar:na] > at > org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) > ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) > [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) > [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] > at > org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.jav
[jira] [Commented] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
[ https://issues.apache.org/jira/browse/DRILL-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15607086#comment-15607086 ] ASF GitHub Bot commented on DRILL-4964: --- GitHub user sohami opened a pull request: https://github.com/apache/drill/pull/628 DRILL-4964: Drill fails to connect to hive metastore after hive metas… …tore is restarted unless drillbits are restarted also Changes: For certain methods in DrillHiveMetaStoreClient like getAllDatabases and getAllTables the HiveMetaStoreClient wraps the exception of type MetaException and TException both into MetaException. So in case of connection failure which is thrown as TException it is difficult to categorize at DrillClient level. Hence the fix is to close older connection and reconnect in case of these 2 api's. In all other cases proper set of exceptions are thrown where we can handle each one individually. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sohami/drill DRILL-4964 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/628.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #628 commit 95477151e9a5b28a121ffe6a879006e4333039c4 Author: Sorabh Hamirwasia Date: 2016-10-26T01:52:01Z DRILL-4964: Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also Changes: For certain methods in DrillHiveMetaStoreClient like getAllDatabases and getAllTables the HiveMetaStoreClient wraps the exception of type MetaException and TException both into MetaException. So in case of connection failure which is thrown as TException it is difficult to categorize at DrillClient level. Hence the fix is to close older connection and reconnect in case of these 2 api's. In all other cases proper set of exceptions are thrown where we can handle each one individually. > Drill fails to connect to hive metastore after hive metastore is restarted > unless drillbits are restarted also > -- > > Key: DRILL-4964 > URL: https://issues.apache.org/jira/browse/DRILL-4964 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0, 1.9.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia > > It is found that if Hive Metastore is restarted then Drillbit also needs to > be restarted to further query and connect to hive metastore. > Repro Steps: > === > 1) Start HiveMetastore and drillbit. > 2) Start Drillbit client with Scheman as hive and run a simple query like > "show databases" >Command to start client: sqlline -u > "jdbc:drill:schema=hive;drillbit=" > 3) restart hive metastore > 4) Execute same query "show databases" on existing drillclient or new one. > You will see that hive default database is not listed. If you query any hive > data then it will fail. > Log snippet from drillbit.log: > == > 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases > 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG > o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin > configuration with zookeeper quorum 'localhost', port '2181'. > 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN > o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access > HiveDatabase 'default'. > java.util.concurrent.ExecutionException: MetaException(message:Got exception: > org.apache.thrift.transport.TTransportException null) > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[guava-18.0.jar:na] > at > com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) > ~[guava-18.0.jar:na] > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320)
[jira] [Updated] (DRILL-4965) query snappy data of hive
[ https://issues.apache.org/jira/browse/DRILL-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rainday updated DRILL-4965: --- Issue Type: Bug (was: Improvement) > query snappy data of hive > - > > Key: DRILL-4965 > URL: https://issues.apache.org/jira/browse/DRILL-4965 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Hive >Affects Versions: 1.8.0 > Environment: CDH 5.2.0 Drill 1.8 >Reporter: rainday > Fix For: Future > > > Currently, I can not query data with snappy format in Hive. > Could anyone help me ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4965) query snappy data of hive
[ https://issues.apache.org/jira/browse/DRILL-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rainday updated DRILL-4965: --- Environment: CDH 5.2.0 Drill 1.8 Fix Version/s: Future Description: Currently, I can not query data with snappy format in Hive. Could anyone help me ? > query snappy data of hive > - > > Key: DRILL-4965 > URL: https://issues.apache.org/jira/browse/DRILL-4965 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Hive >Affects Versions: 1.8.0 > Environment: CDH 5.2.0 Drill 1.8 >Reporter: rainday > Fix For: Future > > > Currently, I can not query data with snappy format in Hive. > Could anyone help me ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4965) query snappy data of hive
rainday created DRILL-4965: -- Summary: query snappy data of hive Key: DRILL-4965 URL: https://issues.apache.org/jira/browse/DRILL-4965 Project: Apache Drill Issue Type: Improvement Components: Storage - Hive Affects Versions: 1.8.0 Reporter: rainday -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4964) Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also
Sorabh Hamirwasia created DRILL-4964: Summary: Drill fails to connect to hive metastore after hive metastore is restarted unless drillbits are restarted also Key: DRILL-4964 URL: https://issues.apache.org/jira/browse/DRILL-4964 Project: Apache Drill Issue Type: Bug Affects Versions: 1.8.0, 1.9.0 Reporter: Sorabh Hamirwasia Assignee: Sorabh Hamirwasia It is found that if Hive Metastore is restarted then Drillbit also needs to be restarted to further query and connect to hive metastore. Repro Steps: === 1) Start HiveMetastore and drillbit. 2) Start Drillbit client with Scheman as hive and run a simple query like "show databases" Command to start client: sqlline -u "jdbc:drill:schema=hive;drillbit=" 3) restart hive metastore 4) Execute same query "show databases" on existing drillclient or new one. You will see that hive default database is not listed. If you query any hive data then it will fail. Log snippet from drillbit.log: == 2016-10-25 18:32:00,561 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 27eff86e-e8fb-3d91-eb88-4af75ff6d174: show databases 2016-10-25 18:32:00,563 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] DEBUG o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase StoragePlugin configuration with zookeeper quorum 'localhost', port '2181'. 2016-10-25 18:32:00,595 [27eff86e-e8fb-3d91-eb88-4af75ff6d174:foreman] WARN o.a.d.e.s.h.schema.HiveSchemaFactory - Failure while attempting to access HiveDatabase 'default'. java.util.concurrent.ExecutionException: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null) at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na] at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na] at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getDatabases(DrillHiveMetaStoreClient.java:415) ~[drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSubSchema(HiveSchemaFactory.java:139) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.(HiveSchemaFactory.java:133) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory.registerSchemas(HiveSchemaFactory.java:118) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.hive.HiveStoragePlugin.registerSchemas(HiveStoragePlugin.java:100) [drill-storage-hive-core-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:61) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:147) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:137) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema(QueryContext.java:123) [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:62) [
[jira] [Updated] (DRILL-4792) Include session options used for a query as part of the profile
[ https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4792: --- Assignee: Arina Ielchiieva (was: Sudheesh Katkam) > Include session options used for a query as part of the profile > --- > > Key: DRILL-4792 > URL: https://issues.apache.org/jira/browse/DRILL-4792 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.9.0 > > Attachments: no_session_options.JPG, session_options_block.JPG, > session_options_collapsed.JPG, session_options_json.JPG > > > Include session options used for a query as part of the profile. > This will be very useful for debugging/diagnostics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[ https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4604: --- Assignee: Arina Ielchiieva (was: Sudheesh Katkam) > Generate warning on Web UI if drillbits version mismatch is detected > > > Key: DRILL-4604 > URL: https://issues.apache.org/jira/browse/DRILL-4604 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > Attachments: index_page.JPG, index_page_mismatch.JPG, > screenshots_with_different_states.docx > > > Display drillbit version on web UI. If any of drillbits version doesn't match > with current drillbit, generate warning. > Screenshots - screenshots_with_different_states.docx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4884) Drill produced IOB exception while querying data of 65536 limitation using non batched reader
[
https://issues.apache.org/jira/browse/DRILL-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606866#comment-15606866
]
ASF GitHub Bot commented on DRILL-4884:
---
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/584
The fix looks fine to me. But do you plan to add a unit test case? thx.
> Drill produced IOB exception while querying data of 65536 limitation using
> non batched reader
> -
>
> Key: DRILL-4884
> URL: https://issues.apache.org/jira/browse/DRILL-4884
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.8.0
> Environment: CentOS 6.5 / JAVA 8
>Reporter: Hongze Zhang
>Assignee: Jinfeng Ni
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Drill produces IOB while using a non batched scanner and limiting SQL by
> 65536.
> SQL:
> {noformat}
> select id from xx limit 1 offset 65535
> {noformat}
> Result:
> {noformat}
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324)
> [classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [classes/:na]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [classes/:na]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [classes/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.lang.IndexOutOfBoundsException: index: 131072, length: 2
> (expected: range(0, 131072))
> at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:175)
> ~[classes/:4.0.27.Final]
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:197)
> ~[classes/:4.0.27.Final]
> at io.netty.buffer.DrillBuf.setChar(DrillBuf.java:517)
> ~[classes/:4.0.27.Final]
> at
> org.apache.drill.exec.record.selection.SelectionVector2.setIndex(SelectionVector2.java:79)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(LimitRecordBatch.java:167)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:145)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
> ~[classes/:na]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
> ~[classes/:na]
> at
> org.apache.drill.exec.physical.impl.BaseRootExec
[jira] [Closed] (DRILL-4879) Hive default storage plugin template shows "null" for fresh install
[
https://issues.apache.org/jira/browse/DRILL-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam closed DRILL-4879.
--
Resolution: Won't Fix
> Hive default storage plugin template shows "null" for fresh install
> ---
>
> Key: DRILL-4879
> URL: https://issues.apache.org/jira/browse/DRILL-4879
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Krystal
>Assignee: Sudheesh Katkam
>
> Deleted the /drill folder from zookeeper; then install drill-1.8.
> [root@mfs41 drill-1.8.0]# cat git.properties
> #Generated by Git-Commit-Id-Plugin
> #Thu Sep 01 20:58:58 UTC 2016
> git.commit.id.abbrev=3b5bedd
> git.commit.user.email=prog...@maprtech.com
> git.commit.message.full=One additional fix fo the DoY unit tests.\n
> git.commit.id=3b5beddce3c1850eaba57aea2d8e7b7d22e641d6
> git.commit.message.short=One additional fix fo the DoY unit tests.
> git.commit.user.name=Paul Rogers
> From the UI, open up the Hive storage plugin. The content of the plugin
> showed as "null" instead of the default template. Below is the snippit of
> the error in drillbit.log that gets generated when I clicked on the "Hive"
> storage plugin:
> {code}
> 2016-09-06 15:08:19,581 [qtp1078495730-50] INFO
> o.a.d.e.server.rest.StorageResources - Failure while trying to access storage
> config: hive
> org.apache.drill.common.exceptions.ExecutionSetupException: Failure setting
> up new storage plugin configuration for config
> org.apache.drill.exec.store.hive.HiveStoragePluginConfig@4adf869f
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:325)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:213)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:263)
> ~[drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.server.rest.StorageResources.getStoragePluginJSON(StorageResources.java:101)
> [drill-java-exec-1.8.0.jar:1.8.0]
> at
> org.apache.drill.exec.server.rest.StorageResources.getStoragePlugin(StorageResources.java:115)
> [drill-java-exec-1.8.0.jar:1.8.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_45]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_45]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_45]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
> [jersey-server-2.8.jar:na]
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269)
> [jersey-server-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> [jersey-common-2.8.jar:na]
> at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
> [jersey-common-2.8.jar:na]
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
> [jersey-common-
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[ https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606169#comment-15606169 ] ASF GitHub Bot commented on DRILL-4604: --- Github user sudheeshkatkam commented on a diff in the pull request: https://github.com/apache/drill/pull/482#discussion_r84969332 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java --- @@ -17,6 +17,7 @@ */ package org.apache.drill.exec; +import org.apache.drill.common.util.DrillVersionInfo; --- End diff -- unused > Generate warning on Web UI if drillbits version mismatch is detected > > > Key: DRILL-4604 > URL: https://issues.apache.org/jira/browse/DRILL-4604 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Sudheesh Katkam > Fix For: Future > > Attachments: index_page.JPG, index_page_mismatch.JPG, > screenshots_with_different_states.docx > > > Display drillbit version on web UI. If any of drillbits version doesn't match > with current drillbit, generate warning. > Screenshots - screenshots_with_different_states.docx. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606162#comment-15606162
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84975052
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
@GET
@Path("/stats.json")
@Produces(MediaType.APPLICATION_JSON)
- public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits",
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit :
work.getContext().getBits()) {
- String initialized = bit.isInitialized() ? " initialized" : " not
initialized";
- stats.add(new Stat("Bit #" + number, bit.getAddress() +
initialized));
- ++number;
+ public Stats getStatsJSON() {
+final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
+
+final Map props = Maps.newLinkedHashMap();
+props.put("Cluster Version", version);
+props.put("Number of Drillbits", work.getContext().getBits().size());
+CoordinationProtos.DrillbitEndpoint currentEndpoint =
work.getContext().getEndpoint();
+final String address = currentEndpoint.getAddress();
+props.put("Data Port Address", address + ":" +
currentEndpoint.getDataPort());
+props.put("User Port Address", address + ":" +
currentEndpoint.getUserPort());
+props.put("Control Port Address", address + ":" +
currentEndpoint.getControlPort());
+props.put("Maximum Direct Memory", DrillConfig.getMaxDirectMemory());
+
+return new Stats(props, collectDrillbits(version));
+ }
+
+ private Collection collectDrillbits(String version) {
+Set drillbits = Sets.newTreeSet();
+for (CoordinationProtos.DrillbitEndpoint endpoint :
work.getContext().getBits()) {
+ boolean versionMatch = version.equals(endpoint.getVersion());
+ DrillbitInfo drillbit = new DrillbitInfo(endpoint.getAddress(),
endpoint.isInitialized(), endpoint.getVersion(), versionMatch);
--- End diff --
Older versions will return `null` for `endpoint.getVersion()`; how does the
patch handle these cases?
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606167#comment-15606167
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84972014
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
@GET
@Path("/stats.json")
@Produces(MediaType.APPLICATION_JSON)
- public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits",
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit :
work.getContext().getBits()) {
- String initialized = bit.isInitialized() ? " initialized" : " not
initialized";
- stats.add(new Stat("Bit #" + number, bit.getAddress() +
initialized));
- ++number;
+ public Stats getStatsJSON() {
+final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
--- End diff --
`final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION_VALIDATOR);`
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606166#comment-15606166
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84973313
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
@GET
@Path("/stats.json")
@Produces(MediaType.APPLICATION_JSON)
- public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits",
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit :
work.getContext().getBits()) {
- String initialized = bit.isInitialized() ? " initialized" : " not
initialized";
- stats.add(new Stat("Bit #" + number, bit.getAddress() +
initialized));
- ++number;
+ public Stats getStatsJSON() {
+final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
+
+final Map props = Maps.newLinkedHashMap();
+props.put("Cluster Version", version);
+props.put("Number of Drillbits", work.getContext().getBits().size());
+CoordinationProtos.DrillbitEndpoint currentEndpoint =
work.getContext().getEndpoint();
+final String address = currentEndpoint.getAddress();
+props.put("Data Port Address", address + ":" +
currentEndpoint.getDataPort());
+props.put("User Port Address", address + ":" +
currentEndpoint.getUserPort());
+props.put("Control Port Address", address + ":" +
currentEndpoint.getControlPort());
+props.put("Maximum Direct Memory", DrillConfig.getMaxDirectMemory());
+
+return new Stats(props, collectDrillbits(version));
+ }
+
+ private Collection collectDrillbits(String version) {
+Set drillbits = Sets.newTreeSet();
+for (CoordinationProtos.DrillbitEndpoint endpoint :
work.getContext().getBits()) {
+ boolean versionMatch = version.equals(endpoint.getVersion());
+ DrillbitInfo drillbit = new DrillbitInfo(endpoint.getAddress(),
endpoint.isInitialized(), endpoint.getVersion(), versionMatch);
--- End diff --
Remove misleading `endpoint.isInitialized()`. Looks like that is a check if
protobuf object is initialized, and not the drillbit.
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606164#comment-15606164
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84970266
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +210,21 @@ private void javaPropertiesToSystemOptions() {
}
/**
+ * If cluster version is the same as default, update option to current
drillbit version.
+ */
+ private void checkAndUpdateClusterVersionOption(OptionManager
optionManager, DrillbitEndpoint drillbitEndpoint) {
--- End diff --
Doc: rename `optionManager` to `systemOptions`
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606168#comment-15606168
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84976814
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java ---
@@ -207,6 +210,21 @@ private void javaPropertiesToSystemOptions() {
}
/**
+ * If cluster version is the same as default, update option to current
drillbit version.
+ */
+ private void checkAndUpdateClusterVersionOption(OptionManager
optionManager, DrillbitEndpoint drillbitEndpoint) {
+OptionValue versionOption =
optionManager.getOption(ExecConstants.CLUSTER_VERSION);
+OptionValidator validator =
SystemOptionManager.getValidator(ExecConstants.CLUSTER_VERSION);
+if (versionOption.equals(validator.getDefault())) {
+ optionManager.setOption(OptionValue.createOption(
--- End diff --
Add a comment in code about what happens when multiple drillbits try to set
the cluster version (since `getOption` and `setOption` are separate calls).
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606163#comment-15606163
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84973496
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
@GET
@Path("/stats.json")
@Produces(MediaType.APPLICATION_JSON)
- public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits",
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit :
work.getContext().getBits()) {
- String initialized = bit.isInitialized() ? " initialized" : " not
initialized";
- stats.add(new Stat("Bit #" + number, bit.getAddress() +
initialized));
- ++number;
+ public Stats getStatsJSON() {
+final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
--- End diff --
Rename `version` to `clusterVersion`
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected
[
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606165#comment-15606165
]
ASF GitHub Bot commented on DRILL-4604:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/482#discussion_r84975543
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java
---
@@ -55,44 +59,89 @@ public Viewable getStats() {
@GET
@Path("/stats.json")
@Produces(MediaType.APPLICATION_JSON)
- public List getStatsJSON() {
-List stats = Lists.newLinkedList();
-stats.add(new Stat("Number of Drill Bits",
work.getContext().getBits().size()));
-int number = 0;
-for (CoordinationProtos.DrillbitEndpoint bit :
work.getContext().getBits()) {
- String initialized = bit.isInitialized() ? " initialized" : " not
initialized";
- stats.add(new Stat("Bit #" + number, bit.getAddress() +
initialized));
- ++number;
+ public Stats getStatsJSON() {
+final String version =
work.getContext().getOptionManager().getOption(ExecConstants.CLUSTER_VERSION).string_val;
+
+final Map props = Maps.newLinkedHashMap();
+props.put("Cluster Version", version);
+props.put("Number of Drillbits", work.getContext().getBits().size());
+CoordinationProtos.DrillbitEndpoint currentEndpoint =
work.getContext().getEndpoint();
+final String address = currentEndpoint.getAddress();
+props.put("Data Port Address", address + ":" +
currentEndpoint.getDataPort());
+props.put("User Port Address", address + ":" +
currentEndpoint.getUserPort());
+props.put("Control Port Address", address + ":" +
currentEndpoint.getControlPort());
+props.put("Maximum Direct Memory", DrillConfig.getMaxDirectMemory());
+
+return new Stats(props, collectDrillbits(version));
+ }
+
+ private Collection collectDrillbits(String version) {
+Set drillbits = Sets.newTreeSet();
+for (CoordinationProtos.DrillbitEndpoint endpoint :
work.getContext().getBits()) {
+ boolean versionMatch = version.equals(endpoint.getVersion());
+ DrillbitInfo drillbit = new DrillbitInfo(endpoint.getAddress(),
endpoint.isInitialized(), endpoint.getVersion(), versionMatch);
+ drillbits.add(drillbit);
}
-stats.add(new Stat("Data Port Address",
work.getContext().getEndpoint().getAddress() +
- ":" + work.getContext().getEndpoint().getDataPort()));
-stats.add(new Stat("User Port Address",
work.getContext().getEndpoint().getAddress() +
- ":" + work.getContext().getEndpoint().getUserPort()));
-stats.add(new Stat("Control Port Address",
work.getContext().getEndpoint().getAddress() +
- ":" + work.getContext().getEndpoint().getControlPort()));
-stats.add(new Stat("Maximum Direct Memory",
DrillConfig.getMaxDirectMemory()));
-
-return stats;
+return drillbits;
}
@XmlRootElement
- public class Stat {
-private String name;
-private Object value;
+ public static class Stats {
--- End diff --
More descriptive name?
(my bad for naming this `Stat` in the first place)
> Generate warning on Web UI if drillbits version mismatch is detected
>
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
> Fix For: Future
>
> Attachments: index_page.JPG, index_page_mismatch.JPG,
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4884) Drill produced IOB exception while querying data of 65536 limitation using non batched reader
[ https://issues.apache.org/jira/browse/DRILL-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606084#comment-15606084 ] Dechang Gu commented on DRILL-4884: --- When I run a similar query with limit 65536 on 1.9.0 master commit 4ee1d4c, all the drillbits are crashed with SIGSEGV fatal error. Here is what I did: 0: jdbc:drill:schema=dfs.tpchPar100> select l_orderkey, count(*) as cnt from lineitem group by l_orderkey limit 6; +-+--+ | l_orderkey | cnt | +-+--+ | 103069602 | 4| | 103070305 | 1| | 103070470 | 1| | 103072326 | 2| | 103072898 | 7| | 103075718 | 5| +-+--+ 6 rows selected (21.493 seconds) 0: jdbc:drill:schema=dfs.tpchPar100> select l_orderkey, count(*) as cnt from lineitem group by l_orderkey limit 65536; Error: CONNECTION ERROR: Connection /10.10.120.101:51025 <--> ucs-node4.perf.lab/10.10.120.104:31010 (user client) closed unexpectedly. Drillbit down? [Error Id: d65fd84b-6336-4540-afe0-de0b9459dcfe ] (state=,code=0) and in the drillbit.out: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7f8a28735cc6, pid=3997, tid=140229356377856 # # JRE version: OpenJDK Runtime Environment (7.0_65-b17) (build 1.7.0_65-mockbuild_2014_07_16_06_06-b00) # Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) # Problematic frame: # J 3152% C2 org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(I)V (111 bytes) @ 0x7f8a28735cc6 [0x7f8a28735c20+0xa6] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /tmp/jvm-3997/hs_error.log [thread 140229298657024 also had an error] [thread 140230122809088 also had an error] [thread 140229306025728 also had an error] [thread 140229336438528 also had an error] # # If you would like to submit a bug report, please include # instructions on how to reproduce the bug and visit: # http://icedtea.classpath.org/bugzilla # 10:51:34,766 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 10:51:34,766 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 10:51:34,767 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/mapr/drillPerf/drillbit/conf/logback.xml] 10:51:34,769 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath. 10:51:34,769 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/mapr/drill/drill-1.9.0/conf/logback.xml] 10:51:34,769 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/mapr/drillPerf/drillbit/conf/logback.xml] 10:51:34,985 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 10:51:34,994 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender] 10:51:35,007 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE] 10:51:35,074 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@7a674e9b - No compression will be used 10:51:35,074 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@7a674e9b - Large window sizes are not allowed. 10:51:35,074 |-WARN in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@7a674e9b - MaxIndex reduced to 21 10:51:35,085 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 10:51:35,150 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /var/log/drill/drillbit_ucs-node4.perf.lab.log 10:51:35,150 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/var/log/drill/drillbit_ucs-node4.perf.lab.log] 10:51:35,153 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.apache.drill] to false 10:51:35,153 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - org.apache.drill level set to INFO 10:51:35,153 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[org.apache.drill] 10:51:35,154 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to INFO 10:51:35,154 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT] 10:51:35,154 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 10:51:35,155 |-INFO in ch.qos.logback.c
[jira] [Commented] (DRILL-4826) Query against INFORMATION_SCHEMA.TABLES degrades as the number of views increases
[
https://issues.apache.org/jira/browse/DRILL-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606077#comment-15606077
]
ASF GitHub Bot commented on DRILL-4826:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/592#discussion_r84967719
--- Diff:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java
---
@@ -233,6 +236,30 @@ private DrillHiveMetaStoreClient(final HiveConf
hiveConf) throws MetaException {
}
}
+ public static List getTablesByNamesByBulkLoadHelper(
+ final HiveMetaStoreClient mClient, final List tableNames,
final String schemaName,
+ final int bulkSize) {
+final int totalTables = tableNames.size();
+final List tables =
Lists.newArrayList();
+
+// In each round, Drill asks for a sub-list of all the requested tables
+for (int fromIndex = 0; fromIndex < totalTables; fromIndex +=
bulkSize) {
+ final int toIndex = Math.min(fromIndex + bulkSize, totalTables);
+ final List eachBulkofTableNames =
tableNames.subList(fromIndex, toIndex);
+ List eachBulkofTables;
+ // Retries once if the first call to fetch the metadata fails
+ try {
+eachBulkofTables =
--- End diff --
`eachBulkofTables = getTableObjectsByNameHelper(mClient, schemaName,
eachBulkofTableNames);`
> Query against INFORMATION_SCHEMA.TABLES degrades as the number of views
> increases
> -
>
> Key: DRILL-4826
> URL: https://issues.apache.org/jira/browse/DRILL-4826
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Parth Chandra
>Assignee: Padma Penumarthy
>
> Queries against INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.VIEWS slow
> down as the number of views increases.
> BI tools like Tableau issue a query like the following at connection time:
> {code}
> select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from
> INFORMATION_SCHEMA.`TABLES` WHERE TABLE_CATALOG LIKE 'DRILL' ESCAPE '\' AND
> TABLE_SCHEMA <> 'sys' AND TABLE_SCHEMA <> 'INFORMATION_SCHEMA'ORDER BY
> TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME
> {code}
> The time to query the information schema tables degrades as the number of
> views increases. On a test system:
> || Views || Time(secs) ||
> |500 | 6 |
> |1000 | 19 |
> |1500 | 33 |
> This can result in a single connection taking more than a minute to establish.
> The problem occurs because we read the view file for every view and this
> appears to take most of the time.
> Querying information_schema.tables does not, in fact, need to open the view
> file at all, it merely needs to get a listing of the view files. Eliminating
> the view file read will speed up the query tremendously.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4792) Include session options used for a query as part of the profile
[
https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606045#comment-15606045
]
ASF GitHub Bot commented on DRILL-4792:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/551#discussion_r84965675
--- Diff: exec/java-exec/src/main/resources/rest/profile/profile.ftl ---
@@ -107,6 +107,42 @@
FOREMAN: ${model.getProfile().getForeman().getAddress()}
TOTAL FRAGMENTS: ${model.getProfile().getTotalFragments()}
+ <#if (model.getOptionList()?size > 0)>
--- End diff --
For older profiles, I assume `model.getOptionList()` returns `null`. How is
`null` handled?
> Include session options used for a query as part of the profile
> ---
>
> Key: DRILL-4792
> URL: https://issues.apache.org/jira/browse/DRILL-4792
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: no_session_options.JPG, session_options_block.JPG,
> session_options_collapsed.JPG, session_options_json.JPG
>
>
> Include session options used for a query as part of the profile.
> This will be very useful for debugging/diagnostics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4792) Include session options used for a query as part of the profile
[
https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606046#comment-15606046
]
ASF GitHub Bot commented on DRILL-4792:
---
Github user sudheeshkatkam commented on a diff in the pull request:
https://github.com/apache/drill/pull/551#discussion_r84965750
--- Diff: exec/java-exec/src/main/resources/rest/profile/profile.ftl ---
@@ -107,6 +107,42 @@
FOREMAN: ${model.getProfile().getForeman().getAddress()}
TOTAL FRAGMENTS: ${model.getProfile().getTotalFragments()}
+ <#if (model.getOptionList()?size > 0)>
+
+Session Options
+
+
+
+
+
+ Overview
+
+
+
+
+
+
+
+
+ Name
+ Value
+
+
+
+<#list model.getOptionList() as option>
+
+${option.getName()}
+${option.getValue()?c}
--- End diff --
What does `?c` do?
> Include session options used for a query as part of the profile
> ---
>
> Key: DRILL-4792
> URL: https://issues.apache.org/jira/browse/DRILL-4792
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Arina Ielchiieva
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: no_session_options.JPG, session_options_block.JPG,
> session_options_collapsed.JPG, session_options_json.JPG
>
>
> Include session options used for a query as part of the profile.
> This will be very useful for debugging/diagnostics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4963) Issues when overloading Drill native functions with dynamic UDFs
[
https://issues.apache.org/jira/browse/DRILL-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zelaine Fong updated DRILL-4963:
Assignee: Arina Ielchiieva
> Issues when overloading Drill native functions with dynamic UDFs
>
>
> Key: DRILL-4963
> URL: https://issues.apache.org/jira/browse/DRILL-4963
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I created jar file which overloads 3 DRILL native functions
> (LOG(VARCHAR-REQUIRED), CURRENT_DATE(VARCHAR-REQUIRED) and
> ABS(VARCHAR-REQUIRED,VARCHAR-REQUIRED)) and registered it as dynamic UDF.
> If I try to use my functions I will get errors:
> {code:xml}
> SELECT CURRENT_DATE('test') FROM (VALUES(1));
> {code}
> Error: FUNCTION ERROR: CURRENT_DATE does not support operand types (CHAR)
> SQL Query null
> {code:xml}
> SELECT ABS('test','test') FROM (VALUES(1));
> {code}
> Error: FUNCTION ERROR: ABS does not support operand types (CHAR,CHAR)
> SQL Query null
> {code:xml}
> SELECT LOG('test') FROM (VALUES(1));
> {code}
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing
> expression in constant expression evaluator LOG('test'). Errors:
> Error in expression at index -1. Error: Missing function implementation:
> castTINYINT(VARCHAR-REQUIRED). Full expression: UNKNOWN EXPRESSION.
> But if I rerun all this queries after "DrillRuntimeException", they will run
> correctly. It seems that Drill have not updated the function signature before
> that error. Also if I add jar as usual UDF (copy jar to
> /drill_home/jars/3rdparty and restart drillbits), all queries will run
> correctly without errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4831) Running refresh table metadata concurrently randomly fails with JsonParseException
[
https://issues.apache.org/jira/browse/DRILL-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606034#comment-15606034
]
Padma Penumarthy commented on DRILL-4831:
-
Current implementation of metadata cache files do not have any concurrency
control in place. If we have multiple connections trying to read/write
metadata cache files at the same time, nothing is guaranteed. The solution
using tmp files (i.e. writing to a tmp file and atomic rename when done)
resolves the conflict between multiple writers, fixing the problem of metadata
cache files getting corrupted. The caveat is that it is dependent on atomicity
guarantee by underlying file system. MapR FS and Mac OS Extended file system
(local file system I tested with) support atomic rename. However, reads fail
with I/O Error during rename for MapR FS, while they work fine on Mac OS.
> Running refresh table metadata concurrently randomly fails with
> JsonParseException
> --
>
> Key: DRILL-4831
> URL: https://issues.apache.org/jira/browse/DRILL-4831
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.8.0
>Reporter: Rahul Challapalli
>Assignee: Padma Penumarthy
> Attachments: error.log, l_3level.tgz
>
>
> git.commit.id.abbrev=f476eb5
> Just run the below command concurrently from 10 different JDBC connections.
> There is a likelihood that you might encounter the below error
> Extracts from the log
> {code}
> Caused By (java.lang.AssertionError) Internal error: Error while applying
> rule DrillPushProjIntoScan, args
> [rel#189411:LogicalProject.NONE.ANY([]).[](input=rel#189289:Subset#3.ENUMERABLE.ANY([]).[],l_orderkey=$1,dir0=$2,dir1=$3,dir2=$4,l_shipdate=$5,l_extendedprice=$6,l_discount=$7),
> rel#189233:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs,
> metadata_caching_pp, l_3level])]
> org.apache.calcite.util.Util.newInternal():792
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> .
> .
> java.lang.Thread.run():745
> Caused By (org.apache.drill.common.exceptions.DrillRuntimeException)
> com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR,
> code 0)): only regular white space (\r, \n, \t) is allowed between tokens
> at [Source: com.mapr.fs.MapRFsDataInputStream@57a574a8; line: 1, column: 2]
> org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():95
> {code}
> Attached the complete log message and the data set
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4963) Issues when overloading Drill native functions with dynamic UDFs
Roman created DRILL-4963:
Summary: Issues when overloading Drill native functions with
dynamic UDFs
Key: DRILL-4963
URL: https://issues.apache.org/jira/browse/DRILL-4963
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 1.9.0
Reporter: Roman
I created jar file which overloads 3 DRILL native functions
(LOG(VARCHAR-REQUIRED), CURRENT_DATE(VARCHAR-REQUIRED) and
ABS(VARCHAR-REQUIRED,VARCHAR-REQUIRED)) and registered it as dynamic UDF.
If I try to use my functions I will get errors:
{code:xml}
SELECT CURRENT_DATE('test') FROM (VALUES(1));
{code}
Error: FUNCTION ERROR: CURRENT_DATE does not support operand types (CHAR)
SQL Query null
{code:xml}
SELECT ABS('test','test') FROM (VALUES(1));
{code}
Error: FUNCTION ERROR: ABS does not support operand types (CHAR,CHAR)
SQL Query null
{code:xml}
SELECT LOG('test') FROM (VALUES(1));
{code}
Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing
expression in constant expression evaluator LOG('test'). Errors:
Error in expression at index -1. Error: Missing function implementation:
castTINYINT(VARCHAR-REQUIRED). Full expression: UNKNOWN EXPRESSION.
But if I rerun all this queries after "DrillRuntimeException", they will run
correctly. It seems that Drill have not updated the function signature before
that error. Also if I add jar as usual UDF (copy jar to
/drill_home/jars/3rdparty and restart drillbits), all queries will run
correctly without errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2123) Order of columns in the Web UI is wrong when columns are explicitly specified in projection list
[
https://issues.apache.org/jira/browse/DRILL-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam updated DRILL-2123:
---
Assignee: (was: Sudheesh Katkam)
> Order of columns in the Web UI is wrong when columns are explicitly specified
> in projection list
>
>
> Key: DRILL-2123
> URL: https://issues.apache.org/jira/browse/DRILL-2123
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Priority: Critical
> Fix For: Future
>
> Attachments: Screen Shot 2015-01-29 at 4.08.06 PM.png
>
>
> I'm running query:
> {code}
> select c_integer,
>c_bigint,
>nullif(c_integer, c_bigint)
> from `dfs.aggregation`.t1
> order by c_integer
> {code}
> In sqlline I get correct order of columns:
> {code}
> 0: jdbc:drill:schema=dfs> select c_integer, c_bigint, nullif(c_integer,
> c_bigint) from `dfs.aggregation`.t1;
> ++++
> | c_integer | c_bigint | EXPR$2 |
> ++++
> | 451237400 | -3477884857818808320 | 451237400 |
> {code}
> In Web UI - columns are sorted in alphabetical order.
> Screenshot is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4962) Drill registers UDFs from packages which were not included in drill-module.conf
Roman created DRILL-4962: Summary: Drill registers UDFs from packages which were not included in drill-module.conf Key: DRILL-4962 URL: https://issues.apache.org/jira/browse/DRILL-4962 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Affects Versions: 1.8.0 Reporter: Roman Priority: Minor Example of UDF structure: ... package.test ... package.test2 ... package.test3 If I add "package.test2" into the drill-module.conf I will see UDFs from this package after jar registering (as expected). But if I add "package.test" or "package.te" into the drill-module.conf I will see all UDFs from all packages (test, test2, test3) after jar registering. So it seems Drill has specific registration logic for UDFs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-4959) Drill 1.8 not able to connect to S3
[ https://issues.apache.org/jira/browse/DRILL-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604842#comment-15604842 ] Gopal Nagar edited comment on DRILL-4959 at 10/25/16 10:40 AM: --- Hi, Really appreciate your quick response. I have used s3a:// in URL and now it's working fine. Only my concern is that simple select query work fine but when i use and join it doesn't work. Below is sample query and the link where i got this syntax. Could you please advise how to fix this issue ? And can you send me some document where i can find the query syntax. Query - select * from `s3-poc`.`poc1.csv` as a INNER join `s3-poc`.`poc2.csv` as b on a.name = b.name https://abhishek-tiwari.com/post/reflections-on-apache-drill Thanks & Regards, Gopal Nagar was (Author: gopalnaga...@gmail.com): Hi, Really appreciate your quick response. I have used s3a:// in URL and now it's working fine. Only my concern is that simple select query work fine but when i use and join it doesn't work. Below is sample query and the link where i got this syntax. Could you please advise how to fix this issue ? And can you send me some document where i can find the query syntax. Query - select * from `s3-poc`.`poc1.csv` as a INNER join `s3-poc`.`poc2.csv` on a.name = b.name https://abhishek-tiwari.com/post/reflections-on-apache-drill Thanks & Regards, Gopal Nagar > Drill 1.8 not able to connect to S3 > --- > > Key: DRILL-4959 > URL: https://issues.apache.org/jira/browse/DRILL-4959 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Gopal Nagar > > Hi Team, > I have followed below details to integrate Drill with AWS S3. Query keep > running for hours and doesn't display any output (I am querying only 2 row > file from S3). > Reference link > --- > https://abhishek-tiwari.com/post/reflections-on-apache-drill > https://drill.apache.org/docs/s3-storage-plugin/ > Query Format (Tried from UI & CLI) > > select * from `s3`.`hive.csv` LIMIT 10; > select * from `s3`.`bucket_name/hive.csv` LIMIT 10; > After seeing below log, I tried including jets3t-0.9.3.jar in jars directory > but it doesn't fix my problem. > Log Details > -- > 2016-10-24 17:00:02,461 [27f1c1ec-d82e-ba2a-2840-e7104320418f:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27f1c1ec-d82e-ba2a-2840-e7104320418f: select * from `s3`.`hive.csv` LIMIT 10 > 2016-10-24 17:00:02,479 [drill-executor-39] ERROR > o.a.d.exec.server.BootStrapContext - > org.apache.drill.exec.work.foreman.Foreman.run() leaked an exception. > java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-4959) Drill 1.8 not able to connect to S3
[ https://issues.apache.org/jira/browse/DRILL-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604842#comment-15604842 ] Gopal Nagar edited comment on DRILL-4959 at 10/25/16 10:38 AM: --- Hi, Really appreciate your quick response. I have used s3a:// in URL and now it's working fine. Only my concern is that simple select query work fine but when i use and join it doesn't work. Below is sample query and the link where i got this syntax. Could you please advise how to fix this issue ? And can you send me some document where i can find the query syntax. Query - select * from `s3-poc`.`poc1.csv` as a INNER join `s3-poc`.`poc2.csv` on a.name = b.name https://abhishek-tiwari.com/post/reflections-on-apache-drill Thanks & Regards, Gopal Nagar was (Author: gopalnaga...@gmail.com): Hi, Really appreciate your quick response. I have used s3a:// in URL and now it's working fine. Only my concern is that simple select query work fine but when i use and join it doesn't work. Below is sample query and the link where i got this syntax. Could you please advise how to fix this issue ? And can you send me some document where i can find the query syntax. Query - select * from `s3-pocjanssen`.`poc1.csv` as a INNER join `s3-pocjanssen`.`poc2.csv` on a.name = b.name https://abhishek-tiwari.com/post/reflections-on-apache-drill Thanks & Regards, Gopal Nagar > Drill 1.8 not able to connect to S3 > --- > > Key: DRILL-4959 > URL: https://issues.apache.org/jira/browse/DRILL-4959 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Gopal Nagar > > Hi Team, > I have followed below details to integrate Drill with AWS S3. Query keep > running for hours and doesn't display any output (I am querying only 2 row > file from S3). > Reference link > --- > https://abhishek-tiwari.com/post/reflections-on-apache-drill > https://drill.apache.org/docs/s3-storage-plugin/ > Query Format (Tried from UI & CLI) > > select * from `s3`.`hive.csv` LIMIT 10; > select * from `s3`.`bucket_name/hive.csv` LIMIT 10; > After seeing below log, I tried including jets3t-0.9.3.jar in jars directory > but it doesn't fix my problem. > Log Details > -- > 2016-10-24 17:00:02,461 [27f1c1ec-d82e-ba2a-2840-e7104320418f:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27f1c1ec-d82e-ba2a-2840-e7104320418f: select * from `s3`.`hive.csv` LIMIT 10 > 2016-10-24 17:00:02,479 [drill-executor-39] ERROR > o.a.d.exec.server.BootStrapContext - > org.apache.drill.exec.work.foreman.Foreman.run() leaked an exception. > java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4959) Drill 1.8 not able to connect to S3
[ https://issues.apache.org/jira/browse/DRILL-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604842#comment-15604842 ] Gopal Nagar commented on DRILL-4959: Hi, Really appreciate your quick response. I have used s3a:// in URL and now it's working fine. Only my concern is that simple select query work fine but when i use and join it doesn't work. Below is sample query and the link where i got this syntax. Could you please advise how to fix this issue ? And can you send me some document where i can find the query syntax. Query - select * from `s3-pocjanssen`.`poc1.csv` as a INNER join `s3-pocjanssen`.`poc2.csv` on a.name = b.name https://abhishek-tiwari.com/post/reflections-on-apache-drill Thanks & Regards, Gopal Nagar > Drill 1.8 not able to connect to S3 > --- > > Key: DRILL-4959 > URL: https://issues.apache.org/jira/browse/DRILL-4959 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Gopal Nagar > > Hi Team, > I have followed below details to integrate Drill with AWS S3. Query keep > running for hours and doesn't display any output (I am querying only 2 row > file from S3). > Reference link > --- > https://abhishek-tiwari.com/post/reflections-on-apache-drill > https://drill.apache.org/docs/s3-storage-plugin/ > Query Format (Tried from UI & CLI) > > select * from `s3`.`hive.csv` LIMIT 10; > select * from `s3`.`bucket_name/hive.csv` LIMIT 10; > After seeing below log, I tried including jets3t-0.9.3.jar in jars directory > but it doesn't fix my problem. > Log Details > -- > 2016-10-24 17:00:02,461 [27f1c1ec-d82e-ba2a-2840-e7104320418f:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 27f1c1ec-d82e-ba2a-2840-e7104320418f: select * from `s3`.`hive.csv` LIMIT 10 > 2016-10-24 17:00:02,479 [drill-executor-39] ERROR > o.a.d.exec.server.BootStrapContext - > org.apache.drill.exec.work.foreman.Foreman.run() leaked an exception. > java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException -- This message was sent by Atlassian JIRA (v6.3.4#6332)
