[jira] [Commented] (DRILL-4995) Allow lazy init when dynamic UDF support is disabled

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731439#comment-15731439
 ] 

Kunal Khatua commented on DRILL-4995:
-

[~RomanKulyk] Is this JIRA verified and ready to close?

> Allow lazy init when dynamic UDF support is disabled
> 
>
> Key: DRILL-4995
> URL: https://issues.apache.org/jira/browse/DRILL-4995
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> Steps in 2 nodes cluster:
> In 1st node:
> 1. Register jar
> 2. Run function (success)
> 3. Disable dynamic UDF support 
> 4. Run function again (success)
> In 2nd node:
> 5. Try to run function (failed).
> In 1st node the function was initialized before disabling dynamic UDF 
> support. But in 2nd node the function was not initialized. So It seems we 
> need to allow lazy initialization when dynamic UDF support is disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4935) Allow drillbits to advertise a configurable host address to Zookeeper

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-4935:

Reviewer: Khurram Faraaz

[~khfaraaz] please verify this feature. You can ask [~agirish] for any docker 
instances that you might need to mimic this setup.

> Allow drillbits to advertise a configurable host address to Zookeeper
> -
>
> Key: DRILL-4935
> URL: https://issues.apache.org/jira/browse/DRILL-4935
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - RPC
>Affects Versions: 1.8.0
>Reporter: Harrison Mebane
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> There are certain situations, such as running Drill in distributed Docker 
> containers, in which it is desirable to advertise a different hostname to 
> Zookeeper than would be output by INetAddress.getLocalHost().  I propose 
> adding a configuration variable 'drill.exec.rpc.bit.advertised.host' and 
> passing this address to Zookeeper when the configuration variable is 
> populated, otherwise falling back to the present behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4604) Generate warning on Web UI if drillbits version mismatch is detected

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-4604:

Reviewer: Krystal  (was: Paul Rogers)

[~knguyen] Please verify and close when the commit goes in.

> Generate warning on Web UI if drillbits version mismatch is detected
> 
>
> Key: DRILL-4604
> URL: https://issues.apache.org/jira/browse/DRILL-4604
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.10.0
>
> Attachments: NEW_matching_drillbits.JPG, 
> NEW_mismatching_drillbits.JPG, index_page.JPG, index_page_mismatch.JPG, 
> screenshots_with_different_states.docx
>
>
> Display drillbit version on web UI. If any of drillbits version doesn't match 
> with current drillbit, generate warning.
> Screenshots - screenshots_with_different_states.docx.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5094) Assure Comparator to be transitive

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5094:

Reviewer: Abhishek Girish

> Assure Comparator to be transitive
> --
>
> Key: DRILL-5094
> URL: https://issues.apache.org/jira/browse/DRILL-5094
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Chunhui Shi
>Priority: Critical
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> In AssignmentCreator.java, one Comparator could break transitive attribute 
> required for a Comparator implementation and the result is not correct.
> E.g. for:
> long IntPlusOne = 0x8000L;
> [0]=2 * IntPlusOne + 5, [1] = 2* IntPlusOne + 8, [2] = 4 * IntPlusOne + 4,
> the compare results will be like:
> compare([0],[1]) = -3,
> compare([1],[2]) = 4,
> compare([0],[2]) = 1 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5015) As per documentation, when issuing a list of drillbits in the connection string, we always attempt to connect only to the first one

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731386#comment-15731386
 ] 

Kunal Khatua commented on DRILL-5015:
-

[~khfaraaz] Please verify and close. 

> As per documentation, when issuing a list of drillbits in the connection 
> string, we always attempt to connect only to the first one
> ---
>
> Key: DRILL-5015
> URL: https://issues.apache.org/jira/browse/DRILL-5015
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.8.0, 1.9.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sudheesh Katkam
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> When trying to connect to a Drill cluster by specifying more than 1 drillbits 
> to connect to, we always attempt to connect to only the first drillbit.
> As an example, we tested against a pair of drillbits, but we always connect 
> to the first entry in the CSV list by querying for the 'current' drillbit. 
> The remaining entries are never attempted.
> [root@pssc-60 agileSqlPerfTests]# /opt/mapr/drill/drill-1.8.0/bin/sqlline  -u 
>  "jdbc:drill:schema=dfs.tmp;drillbit=pssc-61:31010,pssc-62:31010" -f 
> whereAmI.q  | grep -v logback
> 1/1  select * from sys.drillbits where `current`;
> +-++---++--+
> |hostname | user_port  | control_port  | data_port  | current  |
> +-++---++--+
> | pssc-61.qa.lab  | 31010  | 31011 | 31012  | true |
> +-++---++--+
> 1 row selected (0.265 seconds)
> Closing: org.apache.drill.jdbc.impl.DrillConnectionImpl
> apache drill 1.8.0 
> "a little sql for your nosql"
> This property is meant for use by clients when not wanting to overload the ZK 
> for fetching a list of existing Drillbits, but the behaviour doesn't match 
> the documentation. 
> [Making a Direct Drillbit Connection | 
> https://drill.apache.org/docs/using-the-jdbc-driver/#using-the-jdbc-url-format-for-a-direct-drillbit-connection
>  ]
> We need to randomly shuffle between this list and If an entry in the shuffled 
> list is unreachable, we need to try for the next entry in the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4726) Dynamic UDFs support

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-4726:

Reviewer: Rahul Challapalli

[~rkins] Please verify and close the bug

> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  
> Gist - 
> https://gist.github.com/arina-ielchiieva/a1c4cfa3890145c5ecb1b70a39cbff55#file-dynamicudfssupport-md
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5086) ClassCastException when filter pushdown is used with a bigint or float column and metadata caching.

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731377#comment-15731377
 ] 

Kunal Khatua commented on DRILL-5086:
-

[~rhou] Please verify and close this

> ClassCastException when filter pushdown is used with a bigint or float column 
> and metadata caching.
> ---
>
> Key: DRILL-5086
> URL: https://issues.apache.org/jira/browse/DRILL-5086
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.9.0
>Reporter: Robert Hou
>Assignee: Parth Chandra
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
> Attachments: 0_0_1.parquet, 0_0_2.parquet, 0_0_3.parquet, 
> 0_0_4.parquet, 0_0_5.parquet, drill.parquet_metadata
>
>
> This query results in a ClassCastException when filter pushdown is used with 
> metadata caching.  The bigint column is being compared with an integer value.
>0: jdbc:drill:zk=10.10.100.186:5181/drill/rho> select count(\*) from 
> orders_parts_metadata where bigint_id < 1100;
>Error: SYSTEM ERROR: ClassCastException: java.lang.Integer cannot be cast 
> to java.lang.Long
> To reproduce the problem, put the attached files into a directory.  Then 
> create the metadata:
>refresh table metadata dfs.`path_to_directory`;
> For example, if you put the files in 
> /drill/testdata/filter/orders_parts_metadata, then run this sql command
>refresh table metadata dfs.`/drill/testdata/filter/orders_parts_metadata`;
> A similar problem occurs when a float column is being compared with a double 
> value.
>0: jdbc:drill:zk=10.10.100.186:5181/drill/rho> select count(\*) from 
> orders_parts_metadata where float_id < 1100.0;
>Error: SYSTEM ERROR: ClassCastException
> Also when a timestamp column is being compared with a string.
>0: jdbc:drill:zk=10.10.100.186:5181/drill/rho> select count(\*) from 
> orders_parts_metadata where timestamp_id < '2016-10-13';
>Error: SYSTEM ERROR: ClassCastException: java.lang.Integer cannot be cast 
> to java.lang.Long



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5081) Excessive info level logging introduced in DRILL-4203

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-5081:

Fix Version/s: 1.10.0

> Excessive info level logging introduced in DRILL-4203
> -
>
> Key: DRILL-5081
> URL: https://issues.apache.org/jira/browse/DRILL-5081
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sudheesh Katkam
>Assignee: Vitalii Diravka
> Fix For: 1.10.0
>
>
> Excessive info level logging introduced in 
> [8461d10|https://github.com/apache/drill/commit/8461d10b4fd6ce56361d1d826bb3a38b6dc8473c].
>  A line is printed for every row group being read, and for every metadata 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5070) Code gen: create methods in fixed order to allow test verification

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731202#comment-15731202
 ] 

ASF GitHub Bot commented on DRILL-5070:
---

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/684

DRILL-5070: Code gen: create methods in fixed order to allow test 
verification  

A handy technique in testing is to compare generated code against a 
"golden" copy that defines the expected results. However, at present, Drill 
generates code using the method order returned by Class.getDeclaredMethods, but 
this method makes no guarantee about the order of the methods. The order varies 
from one run to the next. There is some evidence this link that order can vary 
even within a single run, though a quick test was unable to reproduce this case.

The fix is simple, in the SignatureHolder constructor, sort methods by name 
after retrieving them from the class. The sort ensures that method order is 
deterministic. Fortunately, the number of methods is small, so the sort step 
adds little cost.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-rogers/drill DRILL-5070

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #684


commit 63d9b5aaffb6982355c785efe26dfede81c0a66a
Author: Paul Rogers 
Date:   2016-11-28T05:51:02Z

DRILL-5070: Actual fix

The fix: order methods by name in the “signature” so that generated
methods occur in the same order every time.

commit b25bbf5c25c23c196710af0e72b79cc97acff0e9
Author: Paul Rogers 
Date:   2016-11-28T05:52:41Z

DRILL-5070: Test cases

Modifies the existing ExpressionTest to capture generated code and
compare it to a “golden” copy. Provides the mechanism to do the
comparison.

Before the fix, comparisons randomly fail due to random method order.
After the fix, the comparisons consistently work, showing that code is
generated in the same order each time.




> Code gen: create methods in fixed order to allow test verification
> --
>
> Key: DRILL-5070
> URL: https://issues.apache.org/jira/browse/DRILL-5070
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> A handy technique in testing is to compare generated code against a "golden" 
> copy that defines the expected results. However, at present, Drill generates 
> code using the method order returned by {{Class.getDeclaredMethods}}, but 
> this method makes no guarantee about the order of the methods. The order 
> varies from one run to the next. There is some evidence [this 
> link|http://stackoverflow.com/questions/28585843/java-reflection-getdeclaredmethods-in-declared-order-strange-behaviour]
>  that order can vary even within a single run, though a quick test was unable 
> to reproduce this case.
> If method order does indeed vary within a single run, then the order can 
> impact the Drill code cache since it compares the sources from two different 
> generation events to detect duplicate code.
> This issue appeared when attempting to modify tests to capture generated code 
> for comparison to future results. Even a simple generated case from 
> {{ExpressionTest.testBasicExpression()}} that generates {{if(true) then 1 
> else 0 end}} (all constants) produced methods in different orders on each 
> test run.
> The fix is simple, in the {{SignatureHolder}} constructor, sort methods by 
> name after retrieving them from the class. The sort ensures that method order 
> is deterministic. Fortunately, the number of methods is small, so the sort 
> step adds little cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731194#comment-15731194
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/676
  
Failure is a Maven transfer failure. How to reset?


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4982) Hive Queries degrade when queries switch between different formats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731190#comment-15731190
 ] 

ASF GitHub Bot commented on DRILL-4982:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/638#discussion_r91446474
  
--- Diff: 
contrib/storage-hive/core/src/main/codegen/templates/HiveRecordReaders.java ---
@@ -0,0 +1,300 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This template is used to generate different Hive record reader classes 
for different data formats
+ * to avoid JIT profile pullusion. These readers are derived from 
HiveAbstractReader which implements
+ * codes for init and setup stage, but the repeated - and performance 
critical part - next() method is
+ * separately implemented in the classes generated from this template. The 
internal SkipRecordReeader
+ * class is also separated as well due to the same reason.
+ *
+ * As to the performance gain with this change, please refer to:
+ * https://issues.apache.org/jira/browse/DRILL-4982
+ *
+ */
+<@pp.dropOutputFile />
+<#list hiveFormat.map as entry>
+<@pp.changeOutputFile 
name="/org/apache/drill/exec/store/hive/Hive${entry.hiveReader}Reader.java" />
+<#include "/@includes/license.ftl" />
+
+package org.apache.drill.exec.store.hive;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.hive.conf.HiveConf;
+
+import org.apache.hadoop.hive.serde2.SerDeException;
+
+import org.apache.hadoop.mapred.RecordReader;
+<#if entry.hasHeaderFooter == true>
+import org.apache.hadoop.hive.serde2.SerDe;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Queue;
+import java.util.Set;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.serde.serdeConstants;
+
+
+public class Hive${entry.hiveReader}Reader extends HiveAbstractReader {
+
+  Object key;
+<#if entry.hasHeaderFooter == true>
--- End diff --

Free marker allows simple boolean expressions:
`<#if entry.hasHeaderFooter>`


> Hive Queries degrade when queries switch between different formats
> --
>
> Key: DRILL-4982
> URL: https://issues.apache.org/jira/browse/DRILL-4982
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have seen degraded performance by doing these steps:
> 1) generate the repro data:
> python script repro.py as below:
> import string
> import random
>  
> for i in range(3000):
> x1 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x2 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x3 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x4 = 

[jira] [Commented] (DRILL-4982) Hive Queries degrade when queries switch between different formats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731188#comment-15731188
 ] 

ASF GitHub Bot commented on DRILL-4982:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/638#discussion_r91447651
  
--- Diff: 
contrib/storage-hive/core/src/main/codegen/templates/HiveRecordReaders.java ---
@@ -0,0 +1,300 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This template is used to generate different Hive record reader classes 
for different data formats
+ * to avoid JIT profile pullusion. These readers are derived from 
HiveAbstractReader which implements
+ * codes for init and setup stage, but the repeated - and performance 
critical part - next() method is
+ * separately implemented in the classes generated from this template. The 
internal SkipRecordReeader
+ * class is also separated as well due to the same reason.
+ *
+ * As to the performance gain with this change, please refer to:
+ * https://issues.apache.org/jira/browse/DRILL-4982
+ *
+ */
+<@pp.dropOutputFile />
+<#list hiveFormat.map as entry>
+<@pp.changeOutputFile 
name="/org/apache/drill/exec/store/hive/Hive${entry.hiveReader}Reader.java" />
+<#include "/@includes/license.ftl" />
+
+package org.apache.drill.exec.store.hive;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.hive.conf.HiveConf;
+
+import org.apache.hadoop.hive.serde2.SerDeException;
+
+import org.apache.hadoop.mapred.RecordReader;
+<#if entry.hasHeaderFooter == true>
+import org.apache.hadoop.hive.serde2.SerDe;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Queue;
+import java.util.Set;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.serde.serdeConstants;
+
+
+public class Hive${entry.hiveReader}Reader extends HiveAbstractReader {
+
+  Object key;
+<#if entry.hasHeaderFooter == true>
+  SkipRecordsInspector skipRecordsInspector;
+<#else>
+  Object value;
+
+
+  public Hive${entry.hiveReader}Reader(Table table, Partition partition, 
InputSplit inputSplit, List projectedColumns,
+   FragmentContext context, final HiveConf hiveConf,
+   UserGroupInformation proxyUgi) throws 
ExecutionSetupException {
+super(table, partition, inputSplit, projectedColumns, context, 
hiveConf, proxyUgi);
+  }
+
+  public  void internalInit(Properties tableProperties, 
RecordReader reader) {
+
+key = reader.createKey();
+<#if entry.hasHeaderFooter == true>
+skipRecordsInspector = new SkipRecordsInspector(tableProperties, 
reader);
+<#else>
+value = reader.createValue();
+
+
+  }
+  private void readHiveRecordAndInsertIntoRecordBatch(Object 
deSerializedValue, int outputRecordIndex) {
+for (int i = 0; i < selectedStructFieldRefs.size(); i++) {
+  Object hiveValue = finalOI.getStructFieldData(deSerializedValue, 

[jira] [Commented] (DRILL-4982) Hive Queries degrade when queries switch between different formats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731189#comment-15731189
 ] 

ASF GitHub Bot commented on DRILL-4982:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/638#discussion_r91447926
  
--- Diff: 
contrib/storage-hive/core/src/main/codegen/templates/HiveRecordReaders.java ---
@@ -0,0 +1,300 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This template is used to generate different Hive record reader classes 
for different data formats
+ * to avoid JIT profile pullusion. These readers are derived from 
HiveAbstractReader which implements
+ * codes for init and setup stage, but the repeated - and performance 
critical part - next() method is
+ * separately implemented in the classes generated from this template. The 
internal SkipRecordReeader
+ * class is also separated as well due to the same reason.
+ *
+ * As to the performance gain with this change, please refer to:
+ * https://issues.apache.org/jira/browse/DRILL-4982
+ *
+ */
+<@pp.dropOutputFile />
+<#list hiveFormat.map as entry>
+<@pp.changeOutputFile 
name="/org/apache/drill/exec/store/hive/Hive${entry.hiveReader}Reader.java" />
+<#include "/@includes/license.ftl" />
+
+package org.apache.drill.exec.store.hive;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.vector.AllocationHelper;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.hive.conf.HiveConf;
+
+import org.apache.hadoop.hive.serde2.SerDeException;
+
+import org.apache.hadoop.mapred.RecordReader;
+<#if entry.hasHeaderFooter == true>
+import org.apache.hadoop.hive.serde2.SerDe;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import com.google.common.collect.Lists;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Queue;
+import java.util.Set;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.serde.serdeConstants;
+
+
+public class Hive${entry.hiveReader}Reader extends HiveAbstractReader {
+
+  Object key;
+<#if entry.hasHeaderFooter == true>
+  SkipRecordsInspector skipRecordsInspector;
+<#else>
+  Object value;
+
+
+  public Hive${entry.hiveReader}Reader(Table table, Partition partition, 
InputSplit inputSplit, List projectedColumns,
+   FragmentContext context, final HiveConf hiveConf,
+   UserGroupInformation proxyUgi) throws 
ExecutionSetupException {
+super(table, partition, inputSplit, projectedColumns, context, 
hiveConf, proxyUgi);
+  }
+
+  public  void internalInit(Properties tableProperties, 
RecordReader reader) {
+
+key = reader.createKey();
+<#if entry.hasHeaderFooter == true>
+skipRecordsInspector = new SkipRecordsInspector(tableProperties, 
reader);
+<#else>
+value = reader.createValue();
+
+
+  }
+  private void readHiveRecordAndInsertIntoRecordBatch(Object 
deSerializedValue, int outputRecordIndex) {
+for (int i = 0; i < selectedStructFieldRefs.size(); i++) {
+  Object hiveValue = finalOI.getStructFieldData(deSerializedValue, 

[jira] [Commented] (DRILL-4982) Hive Queries degrade when queries switch between different formats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731187#comment-15731187
 ] 

ASF GitHub Bot commented on DRILL-4982:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/638#discussion_r91448234
  
--- Diff: contrib/storage-hive/core/src/main/codegen/data/HiveFormats.tdd 
---
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http:# www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{
--- End diff --

Can we explain this a bit? We have 6 reader types. But, the only difference 
in generated code is has header/footer or not. Can we solve the Java 
optimization problem with a "classic" type hierarchy:

```
HiveAbstractReader
. HiveSimpleReader
. . HiveAvroReader
. . ...
. HiveHeaderFooterReader
. . HiveTextReader
. . ...
```

Names are just made up. The point is, can a much simpler Java hierarchy, 
with less duplicated code, solve the problem? If there is one function that is 
sub-optimized, can just that one function be generated in the subclass rather 
than generating duplicate code?


> Hive Queries degrade when queries switch between different formats
> --
>
> Key: DRILL-4982
> URL: https://issues.apache.org/jira/browse/DRILL-4982
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Chunhui Shi
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have seen degraded performance by doing these steps:
> 1) generate the repro data:
> python script repro.py as below:
> import string
> import random
>  
> for i in range(3000):
> x1 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x2 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x3 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x4 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x5 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> x6 = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ 
> in range(random.randrange(19, 27)))
> print 
> "{0}".format(x1),"{0}".format(x2),"{0}".format(x3),"{0}".format(x4),"{0}".format(x5),"{0}".format(x6)
> python repro.py > repro.csv
> 2) put these files in a dfs directory e.g. '/tmp/hiveworkspace/plain'. Under 
> hive prompt, use the following sql command to create an external table:
> CREATE EXTERNAL TABLE `hiveworkspace`.`plain` (`id1` string, `id2` string, 
> `id3` string, `id4` string, `id5` string, `id6` string) ROW FORMAT SERDE 
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' STORED AS TEXTFILE LOCATION 
> '/tmp/hiveworkspace/plain'
> 3) create Hive's table of ORC|PARQUET format:
> CREATE TABLE `hiveworkspace`.`plainorc` STORED AS ORC AS SELECT 
> id1,id2,id3,id4,id5,id6 from `hiveworkspace`.`plain`;
> CREATE TABLE `hiveworkspace`.`plainparquet` STORED AS PARQUET AS SELECT 
> id1,id2,id3,id4,id5,id6 from `hiveworkspace`.`plain`;
> 4) Query switch between these two tables, then the query time on the same 
> table significantly lengthened. On my setup, for ORC, it was 15sec -> 26secs. 
> Queries on table of other formats, after injecting a query to other formats, 
> all have significant slow down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5091) JDBC unit test fail on Java 8

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5091:

Labels: ready-to-commit  (was: )

> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>  Labels: ready-to-commit
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5083) RecordIterator can sometimes restart a query on close

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers updated DRILL-5083:
---
Summary: RecordIterator can sometimes restart a query on close  (was: 
IteratorValidator does not handle RecordIterator cleanup call to next( ))

> RecordIterator can sometimes restart a query on close
> -
>
> Key: DRILL-5083
> URL: https://issues.apache.org/jira/browse/DRILL-5083
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> This one is very confusing...
> In a test with a MergeJoin and external sort, operators are stacked something 
> like this:
> {code}
> Screen
> - MergeJoin
> - - External Sort
> ...
> {code}
> Using the injector to force a OOM in spill, the external sort threw a 
> UserException up the stack. This was handed by:
> {code}
> IteratorValidatorBatchIterator.next( )
> RecordIterator.clearInflightBatches( )
> RecordIterator.close( )
> MergeJoinBatch.close( )
> {code}
> Which does the following:
> {code}
>   // Check whether next() should even have been called in current state.
>   if (null != exceptionState) {
> throw new IllegalStateException(
> {code}
> But, the exceptionState is set, so we end up throwing an 
> IllegalStateException during cleanup.
> Seems the code should agree: if {{next( )}} will be called during cleanup, 
> then {{next( )}} should gracefully handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730812#comment-15730812
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91426330
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -477,18 +477,32 @@ public void testClosedConnectionMethodsThrowRight() {
 }
 
 @Override
+protected boolean isOkayNonthrowingMethod(Method method) {
+  // Java 8 method
+  if ( "getLargeUpdateCount".equals(method.getName())) {
+return true; }
+  return super.isOkayNonthrowingMethod(method);
+}
+
+@Override
 protected boolean isOkaySpecialCaseException(Method method, Throwable 
cause) {
   final boolean result;
   if (super.isOkaySpecialCaseException(method, cause)) {
 result = true;
   }
+  else if (   method.getName().equals("executeLargeBatch")
--- End diff --

This one stays... The style in this file was to align the terms in the 
conditional. Following a pattern established elsewhere in the file.


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730814#comment-15730814
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91426377
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -543,6 +566,14 @@ else if (RuntimeException.class == cause.getClass()
 // Special good-enough case--we had to use RuntimeException for 
now.
 result = true;
   }
+  else if (  method.getName().equals("setObject")
--- End diff --

Same style issue as earlier.


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730813#comment-15730813
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91429339
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/JdbcAssert.java ---
@@ -66,6 +66,11 @@ public static void setFactory(ConnectionFactory factory) 
{
   public static Properties getDefaultProperties() {
 final Properties properties = new Properties();
 properties.setProperty("drillJDBCUnitTests", "true");
+
+// Must set this to false to ensure that the tests ignore any existing
+// plugin configurations stored in /tmp/drill.
+
--- End diff --

I prefer extra lines around comments. Makes the flow less cluttered.


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730667#comment-15730667
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91422620
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -587,6 +618,12 @@ else if (RuntimeException.class == cause.getClass()
 // Special good-enough case--we had to use RuntimeException for 
now.
 result = true;
   }
+  else if (SQLFeatureNotSupportedException.class == cause.getClass()
+  && (   method.getName().equals("updateObject")
--- End diff --

extra space after ( .   move && to line above. 


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730661#comment-15730661
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91423240
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2769UnsupportedReportsUseSqlExceptionTest.java
 ---
@@ -280,6 +280,10 @@ else if (NullPointerException.class == cause.getClass()
   // code implements them.
   successLinesBuf.append(resultLine);
 }
+else if (isOkaySpecialCaseException(method, cause)) {
+  successLinesBuf.append(resultLine);
+}
+
--- End diff --

remove extra line


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730663#comment-15730663
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91421952
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -477,18 +477,32 @@ public void testClosedConnectionMethodsThrowRight() {
 }
 
 @Override
+protected boolean isOkayNonthrowingMethod(Method method) {
+  // Java 8 method
+  if ( "getLargeUpdateCount".equals(method.getName())) {
+return true; }
--- End diff --

remove extra space after if ( .  
} in a separate line.


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730668#comment-15730668
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91421638
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/JdbcAssert.java ---
@@ -66,6 +66,11 @@ public static void setFactory(ConnectionFactory factory) 
{
   public static Properties getDefaultProperties() {
 final Properties properties = new Properties();
 properties.setProperty("drillJDBCUnitTests", "true");
+
+// Must set this to false to ensure that the tests ignore any existing
+// plugin configurations stored in /tmp/drill.
+
--- End diff --

remove the extra line


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730666#comment-15730666
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91421707
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2769UnsupportedReportsUseSqlExceptionTest.java
 ---
@@ -397,10 +419,19 @@ public void testPlainStatementMethodsThrowRight() {
   this.factoryConnection = factoryConnection;
 }
 
+@Override
 protected PreparedStatement getJdbcObject() throws SQLException {
   return factoryConnection.prepareStatement("VALUES 1");
 }
 
+@Override
+protected boolean isOkaySpecialCaseException(Method method,
+ Throwable cause) {
+   // New Java 8 method not supported by Avatica
+
--- End diff --

remove the extra line


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730662#comment-15730662
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91422118
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -477,18 +477,32 @@ public void testClosedConnectionMethodsThrowRight() {
 }
 
 @Override
+protected boolean isOkayNonthrowingMethod(Method method) {
+  // Java 8 method
+  if ( "getLargeUpdateCount".equals(method.getName())) {
+return true; }
+  return super.isOkayNonthrowingMethod(method);
+}
+
+@Override
 protected boolean isOkaySpecialCaseException(Method method, Throwable 
cause) {
   final boolean result;
   if (super.isOkaySpecialCaseException(method, cause)) {
 result = true;
   }
+  else if (   method.getName().equals("executeLargeBatch")
--- End diff --

remove extra space after if. 


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730665#comment-15730665
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91422697
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2769UnsupportedReportsUseSqlExceptionTest.java
 ---
@@ -368,10 +381,19 @@ public void testConnectionMethodsThrowRight() {
   this.factoryConnection = factoryConnection;
 }
 
+@Override
 protected Statement getJdbcObject() throws SQLException {
   return factoryConnection.createStatement();
 }
 
+@Override
+protected boolean isOkaySpecialCaseException(Method method,
+ Throwable cause) {
+   // New Java 8 method not supported by Avatica
+
--- End diff --

remove the extra line


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5091) JDBC unit test fail on Java 8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730664#comment-15730664
 ] 

ASF GitHub Bot commented on DRILL-5091:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/676#discussion_r91422511
  
--- Diff: 
exec/jdbc/src/test/java/org/apache/drill/jdbc/test/Drill2489CallsAfterCloseThrowExceptionsTest.java
 ---
@@ -543,6 +566,14 @@ else if (RuntimeException.class == cause.getClass()
 // Special good-enough case--we had to use RuntimeException for 
now.
 result = true;
   }
+  else if (  method.getName().equals("setObject")
--- End diff --

remove extra space after if


> JDBC unit test fail on Java 8
> -
>
> Key: DRILL-5091
> URL: https://issues.apache.org/jira/browse/DRILL-5091
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
> Environment: Java 8
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Run the {{TestJDBCQuery}} unit tests. They will fail with errors relating to 
> the default name space.
> The problem is due to a failure (that is ignored, DRILL-5090) to set up the 
> test DFS name space.
> The "dfs_test" storage plugin is not found in the plugin registry, resulting 
> in a null object and NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-4347:

Reviewer: Jinfeng Ni  (was: Gautam Kumar Parai)

> Planning time for query64 from TPCDS test suite has increased 10 times 
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0, 
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . >  AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . >  FROM   catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . >  WHERE  cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number = 
> cr_order_number
> . . . . . . . . . . . . >  GROUP  BY cs_item_sk
> . . . . . . . . . . . . >  HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . >  cross_sales
> . . . . . . . . . . . . >  AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk  item_sk,
> . . . . . . . . . . . . > s_store_name   store_name,
> . . . . . . . . . . . . > s_zip  store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number   
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name 
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number   
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name 
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year  AS syear,
> . . . . . . . . . . . . > d2.d_year  AS fsyear,
> . . . . . . . . . . . . > d3.d_year  s2year,
> . . . . . . . . . . . . > Count(*)   cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . >  FROM   store_sales,
> . . . . . . . . . . . . > store_returns,
> . . . . . . . . . . . . > cs_ui,
> . . . . . . . . . . . . > date_dim d1,
> . . . . . . . . . . . . > date_dim d2,
> . . . . . . . . . . . . > date_dim d3,
> . . . . . . . . . . . . > store,
> . . . . . . . . . . . . > customer,
> . . . . . . . . . . . . > customer_demographics cd1,
> . . . . . . . . . . . . > customer_demographics cd2,
> . . . . . . . . . . . . > promotion,
> . . . . . . . . . . . . > household_demographics hd1,
> . . . . . . . . . . . . > household_demographics hd2,
> . . . . . . . . . . . . > customer_address ad1,
> . . . . . . . . . . . . > customer_address ad2,
> . . . . . . . . . . . . > income_band ib1,
> . . . . . . . . . . . . > income_band ib2,
> . . . . . . . . . . . . > item
> . . . . . . . . . . . . >  WHERE  ss_store_sk = s_store_sk
> . . . . . . . . . . . . > AND ss_sold_date_sk = d1.d_date_sk
> . . . . . . . . . . . . > AND ss_customer_sk = c_customer_sk
> . . . . . . . . . . . . > AND ss_cdemo_sk = cd1.cd_demo_sk
> . . . . . . . . . . . . > AND ss_hdemo_sk = hd1.hd_demo_sk
> . . . . . . . . . . . . > AND ss_addr_sk = ad1.ca_address_sk
> . . . . . . . . . . . . 

[jira] [Commented] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730246#comment-15730246
 ] 

ASF GitHub Bot commented on DRILL-4347:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/671
  
Agreed with your comment about the importance of doing calcite rebasing.  
AFAIK, someone(s) else in Ukraine have been working on calcite rebasing for a 
while. Last time I heard is they managed to get a rebased calcite branch and 
are dealing with regressions on Drill side.
 

 


> Planning time for query64 from TPCDS test suite has increased 10 times 
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0, 
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . >  AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . >  FROM   catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . >  WHERE  cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number = 
> cr_order_number
> . . . . . . . . . . . . >  GROUP  BY cs_item_sk
> . . . . . . . . . . . . >  HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . >  cross_sales
> . . . . . . . . . . . . >  AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk  item_sk,
> . . . . . . . . . . . . > s_store_name   store_name,
> . . . . . . . . . . . . > s_zip  store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number   
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name 
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number   
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name 
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year  AS syear,
> . . . . . . . . . . . . > d2.d_year  AS fsyear,
> . . . . . . . . . . . . > d3.d_year  s2year,
> . . . . . . . . . . . . > Count(*)   cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . >  FROM   store_sales,
> . . . . . . . . . . . . > store_returns,
> . . . . . . . . . . . . > cs_ui,
> . . . . . . . . . . . . > date_dim d1,
> . . . . . . . . . . . . > date_dim d2,
> . . . . . . . . . . . . > date_dim d3,
> . . . . . . . . . . . . > store,
> . . . . . . . . . . . . > customer,
> . . . . . . . . . . . . > customer_demographics cd1,
> . . . . . . . . . . . . > customer_demographics cd2,
> . . . . . . . . . . . . > promotion,
> . . . . . . . . . . . . > household_demographics hd1,
> . . . . . . . . . . . . > household_demographics hd2,
> . . . . . . . . . . . . > customer_address ad1,
> . . . . . . . . . . . . > customer_address ad2,
> . . . . . . . . . . . . > income_band ib1,
> . . . . . . . . . . . . > income_band ib2,
> . . . . . . . . . . . . > item
> . . . . . . . . . . . . >  WHERE  ss_store_sk = s_store_sk
> . . . . . . . . . . . . >

[jira] [Commented] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1572#comment-1572
 ] 

ASF GitHub Bot commented on DRILL-4347:
---

Github user julianhyde commented on the issue:

https://github.com/apache/drill/pull/671
  
Rebasing onto Calcite is like running after a train: you can't just stop 
and take a rest. :)

And by the way, the state of Drill-Arrow integration makes me very sad. Now 
Drill has fallen behind there, I doubt whether it will ever catch up.


> Planning time for query64 from TPCDS test suite has increased 10 times 
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0, 
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . >  AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . >  FROM   catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . >  WHERE  cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number = 
> cr_order_number
> . . . . . . . . . . . . >  GROUP  BY cs_item_sk
> . . . . . . . . . . . . >  HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . >  cross_sales
> . . . . . . . . . . . . >  AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk  item_sk,
> . . . . . . . . . . . . > s_store_name   store_name,
> . . . . . . . . . . . . > s_zip  store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number   
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name 
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number   
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name 
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year  AS syear,
> . . . . . . . . . . . . > d2.d_year  AS fsyear,
> . . . . . . . . . . . . > d3.d_year  s2year,
> . . . . . . . . . . . . > Count(*)   cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . >  FROM   store_sales,
> . . . . . . . . . . . . > store_returns,
> . . . . . . . . . . . . > cs_ui,
> . . . . . . . . . . . . > date_dim d1,
> . . . . . . . . . . . . > date_dim d2,
> . . . . . . . . . . . . > date_dim d3,
> . . . . . . . . . . . . > store,
> . . . . . . . . . . . . > customer,
> . . . . . . . . . . . . > customer_demographics cd1,
> . . . . . . . . . . . . > customer_demographics cd2,
> . . . . . . . . . . . . > promotion,
> . . . . . . . . . . . . > household_demographics hd1,
> . . . . . . . . . . . . > household_demographics hd2,
> . . . . . . . . . . . . > customer_address ad1,
> . . . . . . . . . . . . > customer_address ad2,
> . . . . . . . . . . . . > income_band ib1,
> . . . . . . . . . . . . > income_band ib2,
> . . . . . . . . . . . . > item
> . . . . . . . . . . . . >  WHERE  ss_store_sk = s_store_sk
> . . . . . . . . . . . . > AND ss_sold_date_sk = 

[jira] [Commented] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729946#comment-15729946
 ] 

ASF GitHub Bot commented on DRILL-4347:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/671#discussion_r91389995
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdDistinctRowCount.java
 ---
@@ -43,6 +48,30 @@ public Double getDistinctRowCount(RelNode rel, 
ImmutableBitSet groupKey, RexNode
 }
   }
 
+  @Override
+  public Double getDistinctRowCount(Join rel, ImmutableBitSet groupKey, 
RexNode predicate) {
+Double count = null;
+if (rel != null) {
+  if (rel instanceof JoinPrel) {
+// for Drill physical joins, don't recompute the distinct row 
count since it was already done
+// during logical planning; retrieve the cached value.
+count = ((JoinPrel)rel).getDistinctRowCount();
+if (count.doubleValue() < 0) {
+  logger.warn("Invalid cached distinct row count for {}; 
recomputing..", rel.getDescription());
+  count = super.getDistinctRowCount(rel, groupKey, predicate);
+}
+  } else {
+count = super.getDistinctRowCount(rel, groupKey, predicate);
--- End diff --

The API of RelMdDistinctRowCount seems to indicate the distinct rowcount 
depends on input of groupKey and predicate. However, the cached value in 
DrillJoinRel does not differentiate based on groupKey / predicate. Will it 
cause issue in the cases this getDistinctRowCount() is called multiple times 
with different groupKey / predicate? 




> Planning time for query64 from TPCDS test suite has increased 10 times 
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0, 
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . >  AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . >  FROM   catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . >  WHERE  cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number = 
> cr_order_number
> . . . . . . . . . . . . >  GROUP  BY cs_item_sk
> . . . . . . . . . . . . >  HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . >  cross_sales
> . . . . . . . . . . . . >  AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk  item_sk,
> . . . . . . . . . . . . > s_store_name   store_name,
> . . . . . . . . . . . . > s_zip  store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number   
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name 
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number   
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name 
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year  AS syear,
> . . . . . . . . . . . . > d2.d_year  AS fsyear,
> . . . . . . . . . . . . > d3.d_year  s2year,
> . . . . . . . . . . . . > Count(*)   cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . >  

[jira] [Commented] (DRILL-4347) Planning time for query64 from TPCDS test suite has increased 10 times compared to 1.4 release

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729938#comment-15729938
 ] 

ASF GitHub Bot commented on DRILL-4347:
---

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/671
  
@jacques-n , CachingRelMetadataProvider provides caching capability per 
meta-method / rel node [1]. Since Drill logical rel (DrillJoinRel) and Drill 
physical rel (JoinPrel) are different rels, CachingRelMetadataProvider probably 
would not help avoiding the first meta data call for the physical rel nodes, 
even the meta data for logical rels are in the cache. 

@julianhyde , I probably once tried to cherry-pick CALCITE-604 to Drill's 
calcite fork, and I aborted that effort after seeing many merging conflicts (If 
I remember correctly).  Since there has been ongoing effort to rebase Drill 
onto latest Calcite,  it might make sense to see if the rebase work could be 
done shortly. At that time, Drill will benefit from CALCITE-604.

 
1. 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/metadata/CachingRelMetadataProvider.java#L113-L120



> Planning time for query64 from TPCDS test suite has increased 10 times 
> compared to 1.4 release
> --
>
> Key: DRILL-4347
> URL: https://issues.apache.org/jira/browse/DRILL-4347
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.5.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: Future
>
> Attachments: 294e9fb9-cdda-a89f-d1a7-b852878926a1.sys.drill_1.4.0, 
> 294ea418-9fb8-3082-1725-74e3cfe38fe9.sys.drill_1.5.0, drill4347_jstack.txt
>
>
> mapr-drill-1.5.0.201602012001-1.noarch.rpm
> {code}
> 0: jdbc:drill:schema=dfs> WITH cs_ui
> . . . . . . . . . . . . >  AS (SELECT cs_item_sk,
> . . . . . . . . . . . . > Sum(cs_ext_list_price) AS sale,
> . . . . . . . . . . . . > Sum(cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit) AS refund
> . . . . . . . . . . . . >  FROM   catalog_sales,
> . . . . . . . . . . . . > catalog_returns
> . . . . . . . . . . . . >  WHERE  cs_item_sk = cr_item_sk
> . . . . . . . . . . . . > AND cs_order_number = 
> cr_order_number
> . . . . . . . . . . . . >  GROUP  BY cs_item_sk
> . . . . . . . . . . . . >  HAVING Sum(cs_ext_list_price) > 2 * Sum(
> . . . . . . . . . . . . > cr_refunded_cash + 
> cr_reversed_charge
> . . . . . . . . . . . . > + cr_store_credit)),
> . . . . . . . . . . . . >  cross_sales
> . . . . . . . . . . . . >  AS (SELECT i_product_name product_name,
> . . . . . . . . . . . . > i_item_sk  item_sk,
> . . . . . . . . . . . . > s_store_name   store_name,
> . . . . . . . . . . . . > s_zip  store_zip,
> . . . . . . . . . . . . > ad1.ca_street_number   
> b_street_number,
> . . . . . . . . . . . . > ad1.ca_street_name 
> b_streen_name,
> . . . . . . . . . . . . > ad1.ca_cityb_city,
> . . . . . . . . . . . . > ad1.ca_zip b_zip,
> . . . . . . . . . . . . > ad2.ca_street_number   
> c_street_number,
> . . . . . . . . . . . . > ad2.ca_street_name 
> c_street_name,
> . . . . . . . . . . . . > ad2.ca_cityc_city,
> . . . . . . . . . . . . > ad2.ca_zip c_zip,
> . . . . . . . . . . . . > d1.d_year  AS syear,
> . . . . . . . . . . . . > d2.d_year  AS fsyear,
> . . . . . . . . . . . . > d3.d_year  s2year,
> . . . . . . . . . . . . > Count(*)   cnt,
> . . . . . . . . . . . . > Sum(ss_wholesale_cost) s1,
> . . . . . . . . . . . . > Sum(ss_list_price) s2,
> . . . . . . . . . . . . > Sum(ss_coupon_amt) s3
> . . . . . . . . . . . . >  FROM   store_sales,
> . . . . . . . . . . . . > store_returns,
> . . . . . . . . . . . . > cs_ui,
> . . . . . . . . . . . . > date_dim d1,
> . . . . . . . . . . . . > date_dim d2,
> . . . . . . . . . . . . > date_dim d3,
> . . . . . . . . . . . . > store,
> . . . . . . . . . . . . > customer,
> . . . . . . . . . . . . > customer_demographics cd1,
> . . . . . . . . . . . . > 

[jira] [Updated] (DRILL-5108) Reduce output from Maven git-commit-id-plugin

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-5108:
-
Labels: ready-to-commit  (was: )

> Reduce output from Maven git-commit-id-plugin
> -
>
> Key: DRILL-5108
> URL: https://issues.apache.org/jira/browse/DRILL-5108
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
>
> The git-commit-id-plugin grabs information from Git to display during a 
> build. It prints many e-mail addresses and other generic project information. 
> As part of the effort to trim down unit test output, we propose to turn off 
> the verbose output from this plugin.
> Specific change:
> {code}
>   
> pl.project13.maven
> git-commit-id-plugin
> ...
> 
>  false
> {code}
> That is, change the verbose setting from true to false.
> In the unlikely event that some build process depends on the verbose output, 
> we can make the setting a configurable parameter, defaulting to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5108) Reduce output from Maven git-commit-id-plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729539#comment-15729539
 ] 

ASF GitHub Bot commented on DRILL-5108:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/680#discussion_r91358221
  
--- Diff: pom.xml ---
@@ -383,7 +383,7 @@
 
 
   dd.MM. '@' HH:mm:ss z
-  true
+  false
--- End diff --

Can this be undone by giving the mvn command line option  -Dverbose  ?



> Reduce output from Maven git-commit-id-plugin
> -
>
> Key: DRILL-5108
> URL: https://issues.apache.org/jira/browse/DRILL-5108
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The git-commit-id-plugin grabs information from Git to display during a 
> build. It prints many e-mail addresses and other generic project information. 
> As part of the effort to trim down unit test output, we propose to turn off 
> the verbose output from this plugin.
> Specific change:
> {code}
>   
> pl.project13.maven
> git-commit-id-plugin
> ...
> 
>  false
> {code}
> That is, change the verbose setting from true to false.
> In the unlikely event that some build process depends on the verbose output, 
> we can make the setting a configurable parameter, defaulting to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5114) Rationalize use of Logback logging in unit tests

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5114:

Assignee: Paul Rogers
Reviewer: Chunhui Shi

Assigned Reviewer to [~cshi]

> Rationalize use of Logback logging in unit tests
> 
>
> Key: DRILL-5114
> URL: https://issues.apache.org/jira/browse/DRILL-5114
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Drill uses Logback as its logger. The logger is used in several to display 
> some test output. Test output is sent to stdout, rather than a log file. 
> Since Drill also uses Logback, that same configuration sends much Drill 
> logging output to stdout as well, cluttering test output.
> Logback requires that that one Logback config file (either logback.xml or 
> hogback-test.xml) exist on the class path. Tests store the config file in the 
> src/test/resources folder of each sub-project.
> These files set the default logging level to debug. While this setting is 
> fine when working with individual tests, the output is overwhelming for bulk 
> test runs.
> The first requested change is to set the default logging level to error.
> The existing config files are usually called "logback.xml." Change the name 
> of test files to "logback-test.xml" to make clear that they are, in fact, 
> test configs.
> The {{exec/java-exec/src/test/resources/logback.xml}} config file is a full 
> version of Drill's production config file. Replace this with a config 
> suitable for testing (that is, the same as other modules.)
> The java-exec project includes a production-like config file in its non-test 
> sources: {{exec/java-exec/src/main/resources/logback.xml}}. Remove this as it 
> is not needed. (Instead, rely on the one shipped in the distribution 
> subsystem, which is the one copied to the Drill distribution.)
> Since Logback complains bitterly (via many log messages) when it cannot find 
> a configuration file (and each sub-module must have its own test 
> configuration), add missing logging configuration files:
> * exec/memory/base/src/test/resources/logback-test.xml
> * logical/src/test/resources/logback-test.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729482#comment-15729482
 ] 

ASF GitHub Bot commented on DRILL-5032:
---

Github user chunhui-shi commented on the issue:

https://github.com/apache/drill/pull/654
  
+1


> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
> ~[protobuf-java-2.5.0.jar:na]
> at 

[jira] [Commented] (DRILL-5114) Rationalize use of Logback logging in unit tests

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729443#comment-15729443
 ] 

Paul Rogers commented on DRILL-5114:


You can reduce some of Maven’s more excessive logging By modifying the Maven 
config file on your system(s). Please see the instructions here: 
https://github.com/paul-rogers/drill/wiki/Test-Logging#test-logging-with-maven

> Rationalize use of Logback logging in unit tests
> 
>
> Key: DRILL-5114
> URL: https://issues.apache.org/jira/browse/DRILL-5114
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Drill uses Logback as its logger. The logger is used in several to display 
> some test output. Test output is sent to stdout, rather than a log file. 
> Since Drill also uses Logback, that same configuration sends much Drill 
> logging output to stdout as well, cluttering test output.
> Logback requires that that one Logback config file (either logback.xml or 
> hogback-test.xml) exist on the class path. Tests store the config file in the 
> src/test/resources folder of each sub-project.
> These files set the default logging level to debug. While this setting is 
> fine when working with individual tests, the output is overwhelming for bulk 
> test runs.
> The first requested change is to set the default logging level to error.
> The existing config files are usually called "logback.xml." Change the name 
> of test files to "logback-test.xml" to make clear that they are, in fact, 
> test configs.
> The {{exec/java-exec/src/test/resources/logback.xml}} config file is a full 
> version of Drill's production config file. Replace this with a config 
> suitable for testing (that is, the same as other modules.)
> The java-exec project includes a production-like config file in its non-test 
> sources: {{exec/java-exec/src/main/resources/logback.xml}}. Remove this as it 
> is not needed. (Instead, rely on the one shipped in the distribution 
> subsystem, which is the one copied to the Drill distribution.)
> Since Logback complains bitterly (via many log messages) when it cannot find 
> a configuration file (and each sub-module must have its own test 
> configuration), add missing logging configuration files:
> * exec/memory/base/src/test/resources/logback-test.xml
> * logical/src/test/resources/logback-test.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-5114) Rationalize use of Logback logging in unit tests

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729443#comment-15729443
 ] 

Paul Rogers edited comment on DRILL-5114 at 12/7/16 6:11 PM:
-

You can reduce some of Maven’s more excessive logging by modifying the Maven 
config file on your system(s). Please see the instructions here: 
https://github.com/paul-rogers/drill/wiki/Test-Logging#test-logging-with-maven


was (Author: paul-rogers):
You can reduce some of Maven’s more excessive logging By modifying the Maven 
config file on your system(s). Please see the instructions here: 
https://github.com/paul-rogers/drill/wiki/Test-Logging#test-logging-with-maven

> Rationalize use of Logback logging in unit tests
> 
>
> Key: DRILL-5114
> URL: https://issues.apache.org/jira/browse/DRILL-5114
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Drill uses Logback as its logger. The logger is used in several to display 
> some test output. Test output is sent to stdout, rather than a log file. 
> Since Drill also uses Logback, that same configuration sends much Drill 
> logging output to stdout as well, cluttering test output.
> Logback requires that that one Logback config file (either logback.xml or 
> hogback-test.xml) exist on the class path. Tests store the config file in the 
> src/test/resources folder of each sub-project.
> These files set the default logging level to debug. While this setting is 
> fine when working with individual tests, the output is overwhelming for bulk 
> test runs.
> The first requested change is to set the default logging level to error.
> The existing config files are usually called "logback.xml." Change the name 
> of test files to "logback-test.xml" to make clear that they are, in fact, 
> test configs.
> The {{exec/java-exec/src/test/resources/logback.xml}} config file is a full 
> version of Drill's production config file. Replace this with a config 
> suitable for testing (that is, the same as other modules.)
> The java-exec project includes a production-like config file in its non-test 
> sources: {{exec/java-exec/src/main/resources/logback.xml}}. Remove this as it 
> is not needed. (Instead, rely on the one shipped in the distribution 
> subsystem, which is the one copied to the Drill distribution.)
> Since Logback complains bitterly (via many log messages) when it cannot find 
> a configuration file (and each sub-module must have its own test 
> configuration), add missing logging configuration files:
> * exec/memory/base/src/test/resources/logback-test.xml
> * logical/src/test/resources/logback-test.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729426#comment-15729426
 ] 

ASF GitHub Bot commented on DRILL-5032:
---

Github user Serhii-Harnyk commented on a diff in the pull request:

https://github.com/apache/drill/pull/654#discussion_r91351247
  
--- Diff: 
contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/schema/TestColumnListCache.java
 ---
@@ -0,0 +1,78 @@
+/*
--- End diff --

Done


> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> 

[jira] [Updated] (DRILL-5112) Unit tests derived from PopUnitTestBase fail in IDE due to config errors

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5112:

Reviewer: Chunhui Shi

Assigned Reviewer to [~cshi]

> Unit tests derived from PopUnitTestBase fail in IDE due to config errors
> 
>
> Key: DRILL-5112
> URL: https://issues.apache.org/jira/browse/DRILL-5112
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>
> Drill provides a wide variety of unit tests. Many derive from 
> {{PopUnitTestBase}} to test the Physical OPerators.
> The tests use a default configuration:
> {code}
> protected static DrillConfig CONFIG;
>   @BeforeClass
>   public static void setup() {
> CONFIG = DrillConfig.create();
>   }
> {code}
> The tests rely on config settings specified in the {{pom.xml}} file (see note 
> below.) When run in Eclipse, no such config exists, so the tests use only the 
> default config. The defaults allow a web server to be started.
> Many tests start multiple Drillbits using the above config. When this occurs, 
> each tries to start a web server. The second one fails because the HTTP port 
> is already in use.
> The solution is to initialize the config using the same settings as used in 
> the {{BaseTestQuery}} test case: the unit tests then work fine in Eclipse.
> As an aside, having multiple ways to set up the Drill config (and other 
> items) leads to much wasted time as each engineer must learn the quirks of 
> each test hierarchy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5113) Upgrade Maven RAT plugin to avoid annoying XML errors

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-5113:

Labels: ready-to-commit  (was: )

> Upgrade Maven RAT plugin to avoid annoying XML errors
> -
>
> Key: DRILL-5113
> URL: https://issues.apache.org/jira/browse/DRILL-5113
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: ready-to-commit
>
> Build Drill with most Maven logging turned off. On every sub-project you will 
> see the following:
> {code}
> Compiler warnings:
>   WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> [INFO] Starting audit...
> Audit done.
> {code}
> The warning is a known issue with Java: 
> http://bugs.java.com/view_bug.do?bug_id=8016153
> The RAT folks seem to have done a patch: version 0.12 of the plugin no longer 
> has the warning. Upgrade Drill's {{pom.xml}} file to use this version instead 
> of the anonymous version currently used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[B Anil Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729026#comment-15729026
 ] 

B Anil Kumar commented on DRILL-4779:
-

Hi [~rkiss]

 [~kam_iitkgp] and I are working on this feature. But this work is going in bit 
slow pace. 


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting text, json and avro 
> message types



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4941:
-
Fix Version/s: (was: 1.9.0)

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728785#comment-15728785
 ] 

Serhii Harnyk commented on DRILL-4941:
--

So all this queries should fail. To execute all this queries you should cast 
nulls to appropriate types. For example queries:
{noformat}
SELECT (CASE WHEN true or cast(null as boolean) then 1 else 0 end) from 
(VALUES(1))
{noformat}
{noformat}
select NULLIF(-1,cast(null as integer))  from (values(1)) as foo
{noformat}
works and returns right result.

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728760#comment-15728760
 ] 

Serhii Harnyk commented on DRILL-4941:
--

Julian Hyde mentioned:
The SQL standard allows NULL values if you can determine the type, e.g.

{code}
  INSERT INTO Emp (empno, commission) VALUES (100, NULL)
{code}

but not naked NULLs:

{code}
  SELECT empno, NULL FROM Emp
{code}

You have to cast them:

{code}
  SELECT empno, CAST(NULL AS INTEGER) From Emp
{code}

You may think that we should allow NULL in places where we can deduce the 
intended type because there is only one overloaded operator that matches. This 
would be the case with say

{code}
  SELECT * FROM Emp
  WHERE empno > 100 OR NULL
{code}

But the SQL standard doesn't allow it. Postgres goes beyond the standard. 
Calcite goes beyond the standard in some cases, and I'd consider going beyond 
it here. But to make a special case just for BOOLEAN doesn't make sense.

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 

[jira] [Comment Edited] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728518#comment-15728518
 ] 

Khurram Faraaz edited comment on DRILL-4941 at 12/7/16 11:28 AM:
-

MapR Drill 1.9.0 commit ID : 4e97735b

NULLIF function returns as error, whereas Postgres returns correct results.
{noformat}
0: jdbc:drill:schema=dfs.tmp> select NULLIF(-1,null)  from (values(1)) as foo;
Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL

SQL Query null

[Error Id: fbf496c1-5021-4929-a2a7-e68b37ffdde1 on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Postgres returns -1
{noformat}
postgres=# select NULLIF(-1,null)  from (values(1)) as foo;
 nullif

 -1
(1 row)
{noformat}

Stack trace from drillbit.log

{noformat}
2016-12-07 11:15:55,269 [27b81093-a998-d593-3332-4cec6da4511e:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
27b81093-a998-d593-3332-4cec6da4511e: select NULLIF(-1,null)  from (values(1)) 
as foo
2016-12-07 11:15:55,285 [27b81093-a998-d593-3332-4cec6da4511e:foreman] INFO  
o.a.d.exec.planner.sql.SqlConverter - User Error Occurred
org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: class 
org.apache.calcite.sql.SqlLiteral: NULL

SQL Query null

[Error Id: fbf496c1-5021-4929-a2a7-e68b37ffdde1 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:178) 
[drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:622)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:123)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:97)
 [drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008) 
[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264) 
[drill-java-exec-1.9.0.jar:1.9.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.UnsupportedOperationException: class 
org.apache.calcite.sql.SqlLiteral: NULL
at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 

[jira] [Commented] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728518#comment-15728518
 ] 

Khurram Faraaz commented on DRILL-4941:
---


{noformat}
0: jdbc:drill:schema=dfs.tmp> select NULLIF(-1,null)  from (values(1)) as foo;
Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL

SQL Query null

[Error Id: fbf496c1-5021-4929-a2a7-e68b37ffdde1 on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Postgres returns -1
{noformat}
postgres=# select NULLIF(-1,null)  from (values(1)) as foo;
 nullif

 -1
(1 row)
{noformat}

Stack trace from drillbit.log

{noformat}
2016-12-07 11:15:55,269 [27b81093-a998-d593-3332-4cec6da4511e:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
27b81093-a998-d593-3332-4cec6da4511e: select NULLIF(-1,null)  from (values(1)) 
as foo
2016-12-07 11:15:55,285 [27b81093-a998-d593-3332-4cec6da4511e:foreman] INFO  
o.a.d.exec.planner.sql.SqlConverter - User Error Occurred
org.apache.drill.common.exceptions.UserException: VALIDATION ERROR: class 
org.apache.calcite.sql.SqlLiteral: NULL

SQL Query null

[Error Id: fbf496c1-5021-4929-a2a7-e68b37ffdde1 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:178) 
[drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:622)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:123)
 [drill-java-exec-1.9.0.jar:1.9.0]
at 
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:97)
 [drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1008) 
[drill-java-exec-1.9.0.jar:1.9.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:264) 
[drill-java-exec-1.9.0.jar:1.9.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.UnsupportedOperationException: class 
org.apache.calcite.sql.SqlLiteral: NULL
at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164) 
~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
 ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
at 

[jira] [Updated] (DRILL-5085) Add / update description for dynamic UDFs directories in drill-env.sh and drill-module.conf

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5085:

Description: 
1. Add description for $DRILL_TMP_DIR in drill-env.sh
2. Update description for dynamic UDFs directories in drill-module.conf
3. Add dynamic UDFs settings in drill-override-example.conf
4. Add additional logging during udf areas creation.

  was:
1. Add description for $DRILL_TMP_DIR in drill-env.sh
2. Update description for dynamic UDFs directories in drill-module.conf


> Add / update description for dynamic UDFs directories in drill-env.sh and 
> drill-module.conf
> ---
>
> Key: DRILL-5085
> URL: https://issues.apache.org/jira/browse/DRILL-5085
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> 1. Add description for $DRILL_TMP_DIR in drill-env.sh
> 2. Update description for dynamic UDFs directories in drill-module.conf
> 3. Add dynamic UDFs settings in drill-override-example.conf
> 4. Add additional logging during udf areas creation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5085) Add / update description for dynamic UDFs directories in drill-env.sh and drill-module.conf

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728449#comment-15728449
 ] 

Arina Ielchiieva commented on DRILL-5085:
-

Approved by Paul in PR. 

> Add / update description for dynamic UDFs directories in drill-env.sh and 
> drill-module.conf
> ---
>
> Key: DRILL-5085
> URL: https://issues.apache.org/jira/browse/DRILL-5085
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
>  Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> 1. Add description for $DRILL_TMP_DIR in drill-env.sh
> 2. Update description for dynamic UDFs directories in drill-module.conf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[Roman Kiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728160#comment-15728160
 ] 

Roman Kiss commented on DRILL-4779:
---

Hello, just checking if there is a plan to add support for Kafka storage plugin.

> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting text, json and avro 
> message types



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)