[jira] [Commented] (DRILL-4038) [MongoDB] Exception on bson mixin

[Bhallamudi Venkata Siva Kamesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006820#comment-15006820
 ] 

Bhallamudi Venkata Siva Kamesh commented on DRILL-4038:
---

When the property store.mongo.read_numbers_as_double is *false*
{code}
0: jdbc:drill:zk=localhost:2181> ALTER SYSTEM SET 
`store.mongo.read_numbers_as_double` = false;
+---+--+
|  ok   |   summary|
+---+--+
| true  | store.mongo.read_numbers_as_double updated.  |
+---+--+
1 row selected (0.28 seconds)

0: jdbc:drill:zk=localhost:2181> select * from input;
Error: SYSTEM ERROR: IllegalArgumentException: You tried to write a BigInt type 
when you are using a ValueWriter of type NullableFloat8WriterImpl.

Fragment 0:0
[Error Id: ad521f97-a1ef-4ba4-8beb-67470109b42f on localhost:31010] 
(state=,code=0)
{code}

When the property store.mongo.read_numbers_as_double is *true*
{code}
0: jdbc:drill:zk=localhost:2181> ALTER SYSTEM SET 
`store.mongo.read_numbers_as_double` = true;
+---+--+
|  ok   |   summary|
+---+--+
| true  | store.mongo.read_numbers_as_double updated.  |
+---+--+
1 row selected (0.091 seconds)

0: jdbc:drill:zk=localhost:2181> select * from input;
+--+-+
| _id  | MoyLignesPreparees  |
+--+-+
| {"$oid":"5622bcebf9f16d8b892d498e"}  | 0.0 |
| {"$oid":"5622bcebf9f16d8b892d498d"}  | 0.5 |
| {"$oid":"5622bcebf9f16d8b892d498f"}  | 100.0   |
| {"$oid":"5622bcebf9f16d8b892d498a"}  | 200.0   |
+--+-+
4 rows selected (0.143 seconds)
{code}

Closing this as *Not A Problem*. Please feel free to reopen this, incase if you 
are still having issues

> [MongoDB] Exception on bson mixin
> -
>
> Key: DRILL-4038
> URL: https://issues.apache.org/jira/browse/DRILL-4038
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MongoDB
>Affects Versions: 1.2.0
> Environment: Drill running on Linux Suse 2.6
>Reporter: W.
>
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> IllegalArgumentException: You tried to write a BigInt type when you are using 
> a ValueWriter of type NullableFloat8WriterImpl. Fragment 0:0 [Error Id: 
> 337c58a4-2f3c-4f12-bb9b-ecd2c58a9b65
> If collection owns data of different bson type 1 (double) and 16 (32 bits 
> integer), I'm getting the given exception
> Data:
> // 2 documents in the collection
> > db.test.find().pretty()
> { "_id" : ObjectId("5622bcebf9f16d8b892d498e"), "MoyLignesPreparees" : 0 }
> { "_id" : ObjectId("5622bcebf9f16d8b892d498d"), "MoyLignesPreparees" : 0.5 }
> // 2 types of data
> > db.test.count({ "MoyLignesPreparees": { $type: 1 } })
> 1
> > db.test.count({ "MoyLignesPreparees": { $type: 16 } })
> 1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-4038) [MongoDB] Exception on bson mixin

[Bhallamudi Venkata Siva Kamesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bhallamudi Venkata Siva Kamesh closed DRILL-4038.
-
Resolution: Not A Problem

> [MongoDB] Exception on bson mixin
> -
>
> Key: DRILL-4038
> URL: https://issues.apache.org/jira/browse/DRILL-4038
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MongoDB
>Affects Versions: 1.2.0
> Environment: Drill running on Linux Suse 2.6
>Reporter: W.
>
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> IllegalArgumentException: You tried to write a BigInt type when you are using 
> a ValueWriter of type NullableFloat8WriterImpl. Fragment 0:0 [Error Id: 
> 337c58a4-2f3c-4f12-bb9b-ecd2c58a9b65
> If collection owns data of different bson type 1 (double) and 16 (32 bits 
> integer), I'm getting the given exception
> Data:
> // 2 documents in the collection
> > db.test.find().pretty()
> { "_id" : ObjectId("5622bcebf9f16d8b892d498e"), "MoyLignesPreparees" : 0 }
> { "_id" : ObjectId("5622bcebf9f16d8b892d498d"), "MoyLignesPreparees" : 0.5 }
> // 2 types of data
> > db.test.count({ "MoyLignesPreparees": { $type: 1 } })
> 1
> > db.test.count({ "MoyLignesPreparees": { $type: 16 } })
> 1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[Aman Sinha (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006861#comment-15006861
 ] 

Aman Sinha commented on DRILL-3765:
---

[~jni] I did not realize this was assigned to me for review...will do that 
soon.  

> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006653#comment-15006653
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

GitHub user k255 opened a pull request:

https://github.com/apache/drill/pull/258

DRILL-4091: Support for additional gis operations in gis contrib module

Support for commonly used gis functions in gis contrib module: relate, 
contains, crosses, intersects, touches, difference, disjoint, equals, overlaps, 
buffer, union, get x coord. of a point, get y coord of a point.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/k255/drill drill-gis-ext

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit 6081158304ab646b1c022f4ec047df0f6cdc5d1c
Author: potocki 
Date:   2015-11-16T13:05:18Z

Support for additional gis operations (relate, contains, touches, union, 
get x y of a point and more)




> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006655#comment-15006655
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user k255 commented on the pull request:

https://github.com/apache/drill/pull/258#issuecomment-157026760
  
this extends DRILL-3914 functionality


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4090) select * into table support

[Ajay Rana (JIRA)]

Ajay Rana created DRILL-4090:


 Summary: select * into table support
 Key: DRILL-4090
 URL: https://issues.apache.org/jira/browse/DRILL-4090
 Project: Apache Drill
  Issue Type: Improvement
  Components: Query Planning & Optimization, SQL Parser
Affects Versions: 1.2.0
 Environment: Drill 1.2
Reporter: Ajay Rana
Priority: Minor


getting parsing error while querying 
select * into dfs.tmp.`high-val-products` from dfs.root.`/json-docs` where 
mvp>1000;
Error: PARSE ERROR: Encountered "into" at line 1, column 10.
it seems Drill wont support select * into table feature which is widely used in 
the SQL world.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[Karol Potocki (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006648#comment-15006648
 ] 

Karol Potocki commented on DRILL-4091:
--

extends DRILL-3914 functionality

> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4091) Support more functions in gis contrib module

[Karol Potocki (JIRA)]

Karol Potocki created DRILL-4091:


 Summary: Support more functions in gis contrib module
 Key: DRILL-4091
 URL: https://issues.apache.org/jira/browse/DRILL-4091
 Project: Apache Drill
  Issue Type: Improvement
  Components: Functions - Drill
Reporter: Karol Potocki


Support for commonly used gis functions in gis contrib module: relate, 
contains, crosses, intersects, touches, difference, disjoint, buffer, union etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4056) Avro deserialization corrupts data

[Bhallamudi Venkata Siva Kamesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006710#comment-15006710
 ] 

Bhallamudi Venkata Siva Kamesh commented on DRILL-4056:
---

Can you add a test case for this?

> Avro deserialization corrupts data
> --
>
> Key: DRILL-4056
> URL: https://issues.apache.org/jira/browse/DRILL-4056
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.3.0
> Environment: Ubuntu 15.04 - Oracle Java
>Reporter: Stefán Baxter
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
> Attachments: test.zip
>
>
> I have an Avro file that support the following data/schema:
> {"field":"some", "classification":{"variant":"Gæst"}}
> When I select 10 rows from this file I get:
> +-+
> |   EXPR$0|
> +-+
> | Gæst|
> | Voksen  |
> | Voksen  |
> | Invitation KIF KBH  |
> | Invitation KIF KBH  |
> | Ordinarie pris KBH  |
> | Ordinarie pris KBH  |
> | Biljetter 200 krBH  |
> | Biljetter 200 krBH  |
> | Biljetter 200 krBH  |
> +-+
> The bug is that the field values are incorrectly de-serialized and the value 
> from the previous row is retained if the subsequent row is shorter.
> The sql query:
> "select s.classification.variant variant from dfs. as s limit 10;"
> That way the  "Ordinarie pris" becomes "Ordinarie pris KBH" because the 
> previous row had the value "Invitation KIF KBH".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007109#comment-15007109
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

GitHub user hnfgns opened a pull request:

https://github.com/apache/drill/pull/260

DRILL-3171: handle concurrent Zookeeper node creation gracefully

Make ZK node creation/updates logic optimistic.
Refactor ZkAbstractStore to eliminate redundant code around node creation 
logic.
Enhanced documentation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hnfgns/incubator-drill DRILL-3171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/260.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #260


commit cf91dc0b123378d94811da8b6c3171f3fd8d714d
Author: Hanifi Gunes 
Date:   2015-11-04T00:24:48Z

DRILL-3171: handle concurrent Zookeeper node creation gracefully




> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Hanifi Gunes
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4092) Support for INTERSECT

[Victoria Markman (JIRA)]

Victoria Markman created DRILL-4092:
---

 Summary: Support for INTERSECT 
 Key: DRILL-4092
 URL: https://issues.apache.org/jira/browse/DRILL-4092
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Victoria Markman






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4090) select * into table support

[Jacques Nadeau (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006972#comment-15006972
 ] 

Jacques Nadeau commented on DRILL-4090:
---

Drill does provide CTAS functionality (CREATE TABLE AS). This is the ANSI SQL 
standard version SELECT INTO. I believe SELECT INTO is the SQL Server approach 
to the same functionality. Is there a reason you need SELECT INTO and can't use 
CTAS?

> select * into table support
> ---
>
> Key: DRILL-4090
> URL: https://issues.apache.org/jira/browse/DRILL-4090
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization, SQL Parser
>Affects Versions: 1.2.0
> Environment: Drill 1.2
>Reporter: Ajay Rana
>Priority: Minor
>
> getting parsing error while querying 
> select * into dfs.tmp.`high-val-products` from dfs.root.`/json-docs` where 
> mvp>1000;
> Error: PARSE ERROR: Encountered "into" at line 1, column 10.
> it seems Drill wont support select * into table feature which is widely used 
> in the SQL world.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[Hanifi Gunes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanifi Gunes updated DRILL-3171:

Assignee: Sudheesh Katkam  (was: Hanifi Gunes)

> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Sudheesh Katkam
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[Hanifi Gunes (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007114#comment-15007114
 ] 

Hanifi Gunes commented on DRILL-3171:
-

[~sudheeshkatkam] please review.

> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Sudheesh Katkam
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4089) Make JSON pretty printing configurable

[Hanifi Gunes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanifi Gunes updated DRILL-4089:

Assignee: Deneche A. Hakim  (was: Hanifi Gunes)

> Make JSON pretty printing configurable
> --
>
> Key: DRILL-4089
> URL: https://issues.apache.org/jira/browse/DRILL-4089
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Hanifi Gunes
>Assignee: Deneche A. Hakim
>
> Currently JSON record writer emits records pretty-printed and there is no way 
> to configure this behavior. This issue proposes to make this configurable via 
> a prettyPrint switch in -storage- execution configuration with default value 
> of true to ensure backward compatibility.
> As a guideline, the following should be used to dictate Drill to emit records 
> in JSON.
> {code:sql}
> alter [session|sytem] set `store.format`='json';
> {code}
> and this new switch should be used to turn off pretty printing:
> {code:sql}
> alter [session|sytem] set `store.json.writer.uglify`='true';
> {code}
> By default, Drill will use system dependent line feed to seperate JSON blobs 
> when pretty printing turned off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4089) Make JSON pretty printing configurable

[Hanifi Gunes (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007115#comment-15007115
 ] 

Hanifi Gunes commented on DRILL-4089:
-

[~adeneche] please review.

> Make JSON pretty printing configurable
> --
>
> Key: DRILL-4089
> URL: https://issues.apache.org/jira/browse/DRILL-4089
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Hanifi Gunes
>Assignee: Deneche A. Hakim
>
> Currently JSON record writer emits records pretty-printed and there is no way 
> to configure this behavior. This issue proposes to make this configurable via 
> a prettyPrint switch in -storage- execution configuration with default value 
> of true to ensure backward compatibility.
> As a guideline, the following should be used to dictate Drill to emit records 
> in JSON.
> {code:sql}
> alter [session|sytem] set `store.format`='json';
> {code}
> and this new switch should be used to turn off pretty printing:
> {code:sql}
> alter [session|sytem] set `store.json.writer.uglify`='true';
> {code}
> By default, Drill will use system dependent line feed to seperate JSON blobs 
> when pretty printing turned off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007101#comment-15007101
 ] 

ASF GitHub Bot commented on DRILL-3765:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/255#discussion_r44964325
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -116,20 +117,32 @@ public DrillSqlWorker(QueryContext context) {
 
   private RuleSet[] getRules(QueryContext context) {
 StoragePluginRegistry storagePluginRegistry = context.getStorage();
-RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
+
+// Ruleset for the case where VolcanoPlanner is used for everything : 
join, filter/project pushdown, partition pruning.
+RuleSet drillLogicalVolOnlyRules = DrillRuleSets.mergedRuleSets(
 DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
 DrillRuleSets.getJoinPermRules(context),
 DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning is done in Hep-LOPT, 
filter/project pushdown and parttion pruning are done in VolcanoPlanner
+RuleSet drillLogicalHepJoinRules = DrillRuleSets.mergedRuleSets(
+DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
+DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning and partition pruning is 
done in Hep, filter/project pushdown are done in VolcanoPlanner
--- End diff --

This comment refers to partition pruning rules but the ruleset does not 
have those..


> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-4090) select * into table support

[Jacques Nadeau (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006972#comment-15006972
 ] 

Jacques Nadeau edited comment on DRILL-4090 at 11/16/15 5:39 PM:
-

Drill does provide CTAS functionality (CREATE TABLE AS). This is the ANSI SQL 
standard version of SELECT INTO. I believe SELECT INTO is the SQL Server 
approach to the same functionality. Is there a reason you need SELECT INTO and 
can't use CTAS?


was (Author: jnadeau):
Drill does provide CTAS functionality (CREATE TABLE AS). This is the ANSI SQL 
standard version SELECT INTO. I believe SELECT INTO is the SQL Server approach 
to the same functionality. Is there a reason you need SELECT INTO and can't use 
CTAS?

> select * into table support
> ---
>
> Key: DRILL-4090
> URL: https://issues.apache.org/jira/browse/DRILL-4090
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization, SQL Parser
>Affects Versions: 1.2.0
> Environment: Drill 1.2
>Reporter: Ajay Rana
>Priority: Minor
>
> getting parsing error while querying 
> select * into dfs.tmp.`high-val-products` from dfs.root.`/json-docs` where 
> mvp>1000;
> Error: PARSE ERROR: Encountered "into" at line 1, column 10.
> it seems Drill wont support select * into table feature which is widely used 
> in the SQL world.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4089) Make JSON pretty printing configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007088#comment-15007088
 ] 

ASF GitHub Bot commented on DRILL-4089:
---

GitHub user hnfgns opened a pull request:

https://github.com/apache/drill/pull/259

DRILL-4089: make JSON pretty printing configurable

Use system dependent line-feed to seperate records when pretty printing is 
turned off
Refactor ExecConstants to eliminate unneeded modifier & accessor 
declarations

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hnfgns/incubator-drill DRILL-4089

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/259.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #259


commit 0b118c9b34a3bcf4a11a40b5cc25862126bd
Author: Hanifi Gunes 
Date:   2015-11-13T23:51:31Z

DRILL-4089: make JSON pretty printing configurable
Use system dependent line-feed to seperate records when pretty printing is 
turned off
refactor ExecConstants to eliminate unneeded modifier & accessor 
declarations




> Make JSON pretty printing configurable
> --
>
> Key: DRILL-4089
> URL: https://issues.apache.org/jira/browse/DRILL-4089
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Hanifi Gunes
>Assignee: Hanifi Gunes
>
> Currently JSON record writer emits records pretty-printed and there is no way 
> to configure this behavior. This issue proposes to make this configurable via 
> a prettyPrint switch in -storage- execution configuration with default value 
> of true to ensure backward compatibility.
> As a guideline, the following should be used to dictate Drill to emit records 
> in JSON.
> {code:sql}
> alter [session|sytem] set `store.format`='json';
> {code}
> and this new switch should be used to turn off pretty printing:
> {code:sql}
> alter [session|sytem] set `store.json.writer.uglify`='true';
> {code}
> By default, Drill will use system dependent line feed to seperate JSON blobs 
> when pretty printing turned off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4089) Make JSON pretty printing configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007166#comment-15007166
 ] 

ASF GitHub Bot commented on DRILL-4089:
---

Github user adeneche commented on the pull request:

https://github.com/apache/drill/pull/259#issuecomment-157142665
  
+1


> Make JSON pretty printing configurable
> --
>
> Key: DRILL-4089
> URL: https://issues.apache.org/jira/browse/DRILL-4089
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Hanifi Gunes
>Assignee: Deneche A. Hakim
>
> Currently JSON record writer emits records pretty-printed and there is no way 
> to configure this behavior. This issue proposes to make this configurable via 
> a prettyPrint switch in -storage- execution configuration with default value 
> of true to ensure backward compatibility.
> As a guideline, the following should be used to dictate Drill to emit records 
> in JSON.
> {code:sql}
> alter [session|sytem] set `store.format`='json';
> {code}
> and this new switch should be used to turn off pretty printing:
> {code:sql}
> alter [session|sytem] set `store.json.writer.uglify`='true';
> {code}
> By default, Drill will use system dependent line feed to seperate JSON blobs 
> when pretty printing turned off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4089) Make JSON pretty printing configurable

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-4089:

Assignee: Hanifi Gunes  (was: Deneche A. Hakim)

> Make JSON pretty printing configurable
> --
>
> Key: DRILL-4089
> URL: https://issues.apache.org/jira/browse/DRILL-4089
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Hanifi Gunes
>Assignee: Hanifi Gunes
>
> Currently JSON record writer emits records pretty-printed and there is no way 
> to configure this behavior. This issue proposes to make this configurable via 
> a prettyPrint switch in -storage- execution configuration with default value 
> of true to ensure backward compatibility.
> As a guideline, the following should be used to dictate Drill to emit records 
> in JSON.
> {code:sql}
> alter [session|sytem] set `store.format`='json';
> {code}
> and this new switch should be used to turn off pretty printing:
> {code:sql}
> alter [session|sytem] set `store.json.writer.uglify`='true';
> {code}
> By default, Drill will use system dependent line feed to seperate JSON blobs 
> when pretty printing turned off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[Hanifi Gunes (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007181#comment-15007181
 ] 

Hanifi Gunes commented on DRILL-3171:
-

[~adeneche] please review.

> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Sudheesh Katkam
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007395#comment-15007395
 ] 

ASF GitHub Bot commented on DRILL-3765:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/255#discussion_r44988307
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -116,20 +117,32 @@ public DrillSqlWorker(QueryContext context) {
 
   private RuleSet[] getRules(QueryContext context) {
 StoragePluginRegistry storagePluginRegistry = context.getStorage();
-RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
+
+// Ruleset for the case where VolcanoPlanner is used for everything : 
join, filter/project pushdown, partition pruning.
+RuleSet drillLogicalVolOnlyRules = DrillRuleSets.mergedRuleSets(
 DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
 DrillRuleSets.getJoinPermRules(context),
 DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning is done in Hep-LOPT, 
filter/project pushdown and parttion pruning are done in VolcanoPlanner
+RuleSet drillLogicalHepJoinRules = DrillRuleSets.mergedRuleSets(
+DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
+DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning and partition pruning is 
done in Hep, filter/project pushdown are done in VolcanoPlanner
--- End diff --

Ok, I see.   
LGTM +1.  


> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4081) Handle schema changes in ExternalSort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007504#comment-15007504
 ] 

ASF GitHub Bot commented on DRILL-4081:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/257


> Handle schema changes in ExternalSort
> -
>
> Key: DRILL-4081
> URL: https://issues.apache.org/jira/browse/DRILL-4081
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Steven Phillips
>Assignee: Steven Phillips
>
> This improvement will make use of the Union vector to handle schema changes. 
> When a new schema appears, the schema will be "merged" with the previous 
> schema. The result will be a new schema that uses Union type to store the 
> columns where this is a type conflict. All of the batches (including the 
> batches that have already arrived) will be coerced into this new schema.
> A new comparison function will be included to handle the comparison of Union 
> type. Comparison of union type will work as follows:
> 1. All numeric types can be mutually compared, and will be compared using 
> Drill implicit cast rules.
> 2. All other types will not be compared against other types, but only among 
> values of the same type.
> 3. There will be an overall precedence of types with regards to ordering. 
> This precedence is not yet defined, but will be as part of the work on this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4094) Respect -DskipTests=true for JDBC plugin tests

[Andrew (JIRA)]

Andrew created DRILL-4094:
-

 Summary: Respect -DskipTests=true for JDBC plugin tests
 Key: DRILL-4094
 URL: https://issues.apache.org/jira/browse/DRILL-4094
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Other
Reporter: Andrew
Assignee: Andrew
Priority: Trivial


The maven config for the JDBC storage plugin does not respect the -DskipTests 
option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[Aman Sinha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Sinha updated DRILL-3765:
--
Assignee: Jinfeng Ni  (was: Aman Sinha)

> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4095) Regression: TPCDS query 30 throws can not plan exception

[Victoria Markman (JIRA)]

Victoria Markman created DRILL-4095:
---

 Summary: Regression: TPCDS query 30 throws can not plan exception
 Key: DRILL-4095
 URL: https://issues.apache.org/jira/browse/DRILL-4095
 Project: Apache Drill
  Issue Type: New Feature
  Components: Query Planning & Optimization
Affects Versions: 1.2.0, 1.3.0
Reporter: Victoria Markman


Fails in: mapr-drill-1.2.0.201510190924-1.noarch.rpm
 mapr-drill-1.3.0.20152348-1.noarch.rpm
  
It looks like this bug was fixed as  part of DRILL-2949 , but has regressed 
since then.

Reproduction with data shipped with drill (TPCH):
{code}
0: jdbc:drill:schema=dfs> select 
. . . . . . . . . . . . > count(*)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > cp.`tpch/nation.parquet` a,
. . . . . . . . . . . . > cp.`tpch/region.parquet` b
. . . . . . . . . . . . > where 
. . . . . . . . . . . . > a.n_regionkey > (select max(b.r_regionkey) 
from cp.`tpch/region.parquet` b where b.r_nationkey = a.n_nationkey);
Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due 
to either a cartesian join or an inequality join
[Error Id: a52ca497-f654-46ba-b1a7-20d1c0147129 on atsqa4-133.qa.lab:31010] 
(state=,code=0)
{code}

If you remove table from the join, query executes:
{code}
0: jdbc:drill:schema=dfs> select 
. . . . . . . . . . . . > count(*)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > cp.`tpch/nation.parquet` a
. . . . . . . . . . . . > --cp.`tpch/region.parquet` b
. . . . . . . . . . . . > where 
. . . . . . . . . . . . > a.n_regionkey > (select max(b.r_regionkey) 
from cp.`tpch/region.parquet` b where b.r_nationkey = a.n_nationkey);
+-+
| EXPR$0  |
+-+
| 0   |
+-+
1 row selected (0.921 seconds)
{code}

This affects TPCDS query 30:

{code}
0: jdbc:drill:schema=dfs> WITH customer_total_return
. . . . . . . . . . . . >  AS (SELECT wr_returning_customer_sk AS 
ctr_customer_sk,
. . . . . . . . . . . . > ca_state AS ctr_state,
. . . . . . . . . . . . > Sum(wr_return_amt)   AS 
ctr_total_return
. . . . . . . . . . . . >  FROM   web_returns,
. . . . . . . . . . . . > date_dim,
. . . . . . . . . . . . > customer_address
. . . . . . . . . . . . >  WHERE  wr_returned_date_sk = d_date_sk
. . . . . . . . . . . . > AND d_year = 2000
. . . . . . . . . . . . > AND wr_returning_addr_sk = 
ca_address_sk
. . . . . . . . . . . . >  GROUP  BY wr_returning_customer_sk,
. . . . . . . . . . . . >ca_state)
. . . . . . . . . . . . > SELECT c_customer_id,
. . . . . . . . . . . . >c_salutation,
. . . . . . . . . . . . >c_first_name,
. . . . . . . . . . . . >c_last_name,
. . . . . . . . . . . . >c_preferred_cust_flag,
. . . . . . . . . . . . >c_birth_day,
. . . . . . . . . . . . >c_birth_month,
. . . . . . . . . . . . >c_birth_year,
. . . . . . . . . . . . >c_birth_country,
. . . . . . . . . . . . >c_login,
. . . . . . . . . . . . >c_email_address,
. . . . . . . . . . . . >c_last_review_date,
. . . . . . . . . . . . >ctr_total_return
. . . . . . . . . . . . > FROM   customer_total_return ctr1,
. . . . . . . . . . . . >customer_address,
. . . . . . . . . . . . >customer
. . . . . . . . . . . . > WHERE  ctr1.ctr_total_return > (SELECT 
Avg(ctr_total_return) * 1.2
. . . . . . . . . . . . > FROM   
customer_total_return ctr2
. . . . . . . . . . . . > WHERE  ctr1.ctr_state 
= ctr2.ctr_state)
. . . . . . . . . . . . >AND ca_address_sk = c_current_addr_sk
. . . . . . . . . . . . >AND ca_state = 'IN'
. . . . . . . . . . . . >AND ctr1.ctr_customer_sk = c_customer_sk
. . . . . . . . . . . . > LIMIT 100;
Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due 
to either a cartesian join or an inequality join
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4047) Select with options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007391#comment-15007391
 ] 

ASF GitHub Bot commented on DRILL-4047:
---

Github user julienledem commented on a diff in the pull request:

https://github.com/apache/drill/pull/246#discussion_r44987917
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import static java.lang.String.format;
+import static org.apache.drill.TestBuilder.listOf;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+
+import org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class TestSelectWithOption extends BaseTestQuery {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(WorkspaceSchemaFactory.class);
+
+  private File genCSVFile(String name, String... rows) throws IOException {
+File file = new File(format("target/%s_%s.csv", 
this.getClass().getName(), name));
+try (FileWriter fw = new FileWriter(file)) {
+  for (int i = 0; i < rows.length; i++) {
+fw.append(rows[i] + "\n");
+  }
+}
+return file;
+  }
+
+  private String genCSVTable(String name, String... rows) throws 
IOException {
+File f = genCSVFile(name, rows);
+return format("dfs.`${WORKING_PATH}/%s`", f.getPath());
+  }
+
+  private void testWithResult(String query, Object... expectedResult) 
throws Exception {
+TestBuilder builder = testBuilder()
+.sqlQuery(query)
+.ordered()
+.baselineColumns("columns");
+for (Object o : expectedResult) {
+  builder = builder.baselineValues(o);
+}
+builder.build().run();
+  }
+
+  @Test
+  public void testTextFieldDelimiter() throws Exception {
+String tableName = genCSVTable("testTextFieldDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+String queryTemplate =
+"select columns from table(%s (type => 'TeXT', fieldDelimiter => 
'%s'))";
+testWithResult(format(queryTemplate, tableName, ","),
+listOf("b\"|\"0"),
+listOf("b\"|\"1"),
+listOf("b\"|\"2")
+  );
+testWithResult(format(queryTemplate, tableName, "|"),
+listOf("b", "0"),
+listOf("b", "1"),
+listOf("b", "2")
+  );
+  }
+
+  @Test @Ignore // It does not look like lineDelimiter is working
+  public void testTextLineDelimiter() throws Exception {
+String tableName = genCSVTable("testTextLineDelimiter",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
lineDelimiter => '|'))", tableName),
+listOf("\"b\""),
+listOf("\"0\"", "\"b\""),
+listOf("\"1\"", "\"b\""),
+listOf("\"2\"")
+  );
+  }
+
+  @Test
+  public void testTextQuote() throws Exception {
+String tableName = genCSVTable("testTextQuote",
+"\"b\"|\"0\"",
+"\"b\"|\"1\"",
+"\"b\"|\"2\"");
+
+testWithResult(format("select columns from table(%s(type => 'TeXT', 
fieldDelimiter => '|', quote => '@'))", tableName),
+listOf("\"b\"", "\"0\""),
+listOf("\"b\"", "\"1\""),
+listOf("\"b\"", "\"2\"")
+);
+
+String quoteTableName = genCSVTable("testTextQuote2",
+"@b@|@0@",
+"@b$@c@|@1@");
+// It seems that a parameter can not be called "escape"
+testWithResult(format("select columns from table(%s(`escape` => '$', 
type => 'TeXT', fieldDelimiter => '|', quote => '@'))", 

[jira] [Commented] (DRILL-4094) Respect -DskipTests=true for JDBC plugin tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007409#comment-15007409
 ] 

ASF GitHub Bot commented on DRILL-4094:
---

GitHub user aleph-zero opened a pull request:

https://github.com/apache/drill/pull/261

DRILL-4094: Respect skipTests in JDBC plugin tests.

Modifies the pom.xml for the JDBC plugin to respect the maven
-DskipTests flag.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aleph-zero/drill issues/DRILL-4094

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #261


commit 98dab7ef851d0929cb5c67dc1c0b8f35b738a8cd
Author: aleph-zero 
Date:   2015-11-16T21:51:31Z

DRILL-4094: Respect skipTests in JDBC plugin tests.

Modifies the pom.xml for the JDBC plugin to respect the maven
-DskipTests flag.




> Respect -DskipTests=true for JDBC plugin tests
> --
>
> Key: DRILL-4094
> URL: https://issues.apache.org/jira/browse/DRILL-4094
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Reporter: Andrew
>Assignee: Andrew
>Priority: Trivial
>
> The maven config for the JDBC storage plugin does not respect the -DskipTests 
> option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4095) Regression: TPCDS query 30 throws can not plan exception

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007539#comment-15007539
 ] 

Victoria Markman commented on DRILL-4095:
-

Note: when bug is fixed, enable 
Advanced/tpcds/tpcds_sf1/original/text/query30.sql.drill-4095.fail

> Regression: TPCDS query 30 throws can not plan exception
> 
>
> Key: DRILL-4095
> URL: https://issues.apache.org/jira/browse/DRILL-4095
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Query Planning & Optimization
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Victoria Markman
>
> Fails in: mapr-drill-1.2.0.201510190924-1.noarch.rpm
>  mapr-drill-1.3.0.20152348-1.noarch.rpm
>   
> It looks like this bug was fixed as  part of DRILL-2949 , but has regressed 
> since then.
> Reproduction with data shipped with drill (TPCH):
> {code}
> 0: jdbc:drill:schema=dfs> select 
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > cp.`tpch/nation.parquet` a,
> . . . . . . . . . . . . > cp.`tpch/region.parquet` b
> . . . . . . . . . . . . > where 
> . . . . . . . . . . . . > a.n_regionkey > (select max(b.r_regionkey) 
> from cp.`tpch/region.parquet` b where b.r_nationkey = a.n_nationkey);
> Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due 
> to either a cartesian join or an inequality join
> [Error Id: a52ca497-f654-46ba-b1a7-20d1c0147129 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> If you remove table from the join, query executes:
> {code}
> 0: jdbc:drill:schema=dfs> select 
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > cp.`tpch/nation.parquet` a
> . . . . . . . . . . . . > --cp.`tpch/region.parquet` b
> . . . . . . . . . . . . > where 
> . . . . . . . . . . . . > a.n_regionkey > (select max(b.r_regionkey) 
> from cp.`tpch/region.parquet` b where b.r_nationkey = a.n_nationkey);
> +-+
> | EXPR$0  |
> +-+
> | 0   |
> +-+
> 1 row selected (0.921 seconds)
> {code}
> This affects TPCDS query 30:
> {code}
> 0: jdbc:drill:schema=dfs> WITH customer_total_return
> . . . . . . . . . . . . >  AS (SELECT wr_returning_customer_sk AS 
> ctr_customer_sk,
> . . . . . . . . . . . . > ca_state AS 
> ctr_state,
> . . . . . . . . . . . . > Sum(wr_return_amt)   AS 
> ctr_total_return
> . . . . . . . . . . . . >  FROM   web_returns,
> . . . . . . . . . . . . > date_dim,
> . . . . . . . . . . . . > customer_address
> . . . . . . . . . . . . >  WHERE  wr_returned_date_sk = d_date_sk
> . . . . . . . . . . . . > AND d_year = 2000
> . . . . . . . . . . . . > AND wr_returning_addr_sk = 
> ca_address_sk
> . . . . . . . . . . . . >  GROUP  BY wr_returning_customer_sk,
> . . . . . . . . . . . . >ca_state)
> . . . . . . . . . . . . > SELECT c_customer_id,
> . . . . . . . . . . . . >c_salutation,
> . . . . . . . . . . . . >c_first_name,
> . . . . . . . . . . . . >c_last_name,
> . . . . . . . . . . . . >c_preferred_cust_flag,
> . . . . . . . . . . . . >c_birth_day,
> . . . . . . . . . . . . >c_birth_month,
> . . . . . . . . . . . . >c_birth_year,
> . . . . . . . . . . . . >c_birth_country,
> . . . . . . . . . . . . >c_login,
> . . . . . . . . . . . . >c_email_address,
> . . . . . . . . . . . . >c_last_review_date,
> . . . . . . . . . . . . >ctr_total_return
> . . . . . . . . . . . . > FROM   customer_total_return ctr1,
> . . . . . . . . . . . . >customer_address,
> . . . . . . . . . . . . >customer
> . . . . . . . . . . . . > WHERE  ctr1.ctr_total_return > (SELECT 
> Avg(ctr_total_return) * 1.2
> . . . . . . . . . . . . > FROM   
> customer_total_return ctr2
> . . . . . . . . . . . . > WHERE  
> ctr1.ctr_state = ctr2.ctr_state)
> . . . . . . . . . . . . >AND ca_address_sk = c_current_addr_sk
> . . . . . . . . . . . . >AND ca_state = 'IN'
> . . . . . . . . . . . . >AND ctr1.ctr_customer_sk = c_customer_sk
> . . . . . . . . . . . . > LIMIT 100;
> Error: UNSUPPORTED_OPERATION ERROR: This query cannot be planned possibly due 
> to either a cartesian join or an inequality join
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-867) tpcds queries 6, 9 and 10 fail to plan

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman resolved DRILL-867.

Resolution: Fixed

> tpcds queries 6, 9 and 10 fail to plan
> --
>
> Key: DRILL-867
> URL: https://issues.apache.org/jira/browse/DRILL-867
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser
>Reporter: Krystal
>Priority: Minor
> Fix For: Future
>
>
> git.commit.id.abbrev=e1e5ea0
> git.commit.time=29.05.2014 @ 15\:32\:29 PDT
> query 6:
> {code}
> select * from (select  a.ca_state state, count(*) cnt
>  from customer_address a
>  ,customer c
>  ,store_sales s
>  ,date_dim d
>  ,item i
>  where  a.ca_address_sk = c.c_current_addr_sk
>   and c.c_customer_sk = s.ss_customer_sk
>   and s.ss_sold_date_sk = d.d_date_sk
>   and s.ss_item_sk = i.i_item_sk
>   and d.d_month_seq = 
>(select distinct (d.d_month_seq)
> from date_dim d
>where d.d_year = 1998
>   and d.d_moy = 5 ) 
>   and i.i_current_price > 1.2 * 
>  (select avg(j.i_current_price)
>from item j 
>where j.i_category = i.i_category)
>  group by a.ca_state
>  having count(*) >= 10
>  order by cnt
>  ) limit 100;
> {code}
> query 7:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007223#comment-15007223
 ] 

ASF GitHub Bot commented on DRILL-3765:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/255#discussion_r44974839
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -116,20 +117,32 @@ public DrillSqlWorker(QueryContext context) {
 
   private RuleSet[] getRules(QueryContext context) {
 StoragePluginRegistry storagePluginRegistry = context.getStorage();
-RuleSet drillLogicalRules = DrillRuleSets.mergedRuleSets(
+
+// Ruleset for the case where VolcanoPlanner is used for everything : 
join, filter/project pushdown, partition pruning.
+RuleSet drillLogicalVolOnlyRules = DrillRuleSets.mergedRuleSets(
 DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
 DrillRuleSets.getJoinPermRules(context),
 DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning is done in Hep-LOPT, 
filter/project pushdown and parttion pruning are done in VolcanoPlanner
+RuleSet drillLogicalHepJoinRules = DrillRuleSets.mergedRuleSets(
+DrillRuleSets.getDrillBasicRules(context),
+DrillRuleSets.getPruneScanRules(context),
+DrillRuleSets.getDrillUserConfigurableLogicalRules(context));
+
+// Ruleset for the case where join planning and partition pruning is 
done in Hep, filter/project pushdown are done in VolcanoPlanner
--- End diff --

Exactly. The ruleset defined here is used in Frameworks.VolcanoPlanner. In 
case that join / partition pruning is done in Hep, we do not put the 
join/partition pruning rules in the ruleset here. In stead, the join/partition 
pruning rules are specified when HepPlanner is built. 


> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Aman Sinha
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[Hanifi Gunes (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanifi Gunes updated DRILL-3171:

Assignee: Deneche A. Hakim  (was: Sudheesh Katkam)

> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-867) tpcds queries 6, 9 and 10 fail to plan

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007330#comment-15007330
 ] 

Victoria Markman commented on DRILL-867:


All these queries throw proper unsupported errors now and have enhancement 
requests associated with the unsupported features:

query 6 (DRILL-3801)
{code}
0: jdbc:drill:schema=dfs> select * from (select  a.ca_state state, count(*) cnt
. . . . . . . . . . . . >  from customer_address a
. . . . . . . . . . . . >  ,customer c
. . . . . . . . . . . . >  ,store_sales s
. . . . . . . . . . . . >  ,date_dim d
. . . . . . . . . . . . >  ,item i
. . . . . . . . . . . . >  where  a.ca_address_sk = c.c_current_addr_sk
. . . . . . . . . . . . >   and c.c_customer_sk = s.ss_customer_sk
. . . . . . . . . . . . >   and s.ss_sold_date_sk = d.d_date_sk
. . . . . . . . . . . . >   and s.ss_item_sk = i.i_item_sk
. . . . . . . . . . . . >   and d.d_month_seq = 
. . . . . . . . . . . . >(select distinct (d.d_month_seq)
. . . . . . . . . . . . > from date_dim d
. . . . . . . . . . . . >where d.d_year = 1998
. . . . . . . . . . . . >   and d.d_moy = 5 ) 
. . . . . . . . . . . . >   and i.i_current_price > 1.2 * 
. . . . . . . . . . . . >  (select avg(j.i_current_price)
. . . . . . . . . . . . >from item j 
. . . . . . . . . . . . >where j.i_category = i.i_category)
. . . . . . . . . . . . >  group by a.ca_state
. . . . . . . . . . . . >  having count(*) >= 10
. . . . . . . . . . . . >  order by cnt
. . . . . . . . . . . . >  ) limit 100;
Error: UNSUPPORTED_OPERATION ERROR: Non-scalar sub-query used in an expression
See Apache Drill JIRA: DRILL-1937
[Error Id: 841dae77-8b74-484e-a0f0-a01a2c07fbd8 on atsqa4-133.qa.lab:31010] 
(state=,code=0)
{code}

query 9 ( DRILL-4092)
{code}
0: jdbc:drill:schema=dfs> select * from (select store.s_store_name
. . . . . . . . . . . . >   ,sum(ss.ss_net_profit)
. . . . . . . . . . . . >  from store_sales ss
. . . . . . . . . . . . >  ,date_dim dd
. . . . . . . . . . . . >  ,store,
. . . . . . . . . . . . >  (select ca_zip
. . . . . . . . . . . . >  from (
. . . . . . . . . . . . >  (SELECT substr(ca.ca_zip,1,5) ca_zip
. . . . . . . . . . . . >   FROM customer_address ca
. . . . . . . . . . . . >   WHERE substr(ca.ca_zip,1,5) IN (
. . . . . . . . . . . . >   
'16733','50732','51878','16933','33177','55974',
. . . . . . . . . . . . >   
'21338','90455','63106','78712','45114',
. . . . . . . . . . . . >   
'51090','44881','35526','91360','34986',
. . . . . . . . . . . . >   
'31893','28853','84061','25483','84541',
. . . . . . . . . . . . >   
'39275','56211','51199','85189','24292',
. . . . . . . . . . . . >   
'27477','46388','77218','21137','43660',
. . . . . . . . . . . . >   
'36509','77925','11691','26790','35256',
. . . . . . . . . . . . >   
'59221','42491','39214','35273','27293',
. . . . . . . . . . . . >   
'74258','68798','50936','19136','25240',
. . . . . . . . . . . . >   
'89163','21667','30941','61680','10425',
. . . . . . . . . . . . >   
'96787','84569','37596','84291','44843',
. . . . . . . . . . . . >   
'31487','24949','31269','62115','79494',
. . . . . . . . . . . . >   
'32194','62531','61655','40724','29091',
. . . . . . . . . . . . >   
'81608','77126','32704','79045','19008',
. . . . . . . . . . . . >   
'81581','59693','24689','79355','19635',
. . . . . . . . . . . . >   
'52025','83585','56103','80150','26203',
. . . . . . . . . . . . >   
'81571','85657','39672','62868','33498',
. . . . . . . . . . . . >   
'69453','25748','44145','35695','57860',
. . . . . . . . . . . . >   
'59532','76967','81235','22004','34487',
. . . . . . . . . . . . >   
'48499','47318','63039','77728','89774',
. . . . . . . . . . . . >   
'91640','76501','70137','37512','48507',
. . . . . . . . . . . . >   
'51980','34851','54884','30905','12745',
. . . . . . . . . . . . >   
'60630','42798','39923','47591','82518',
. . . . . . . . . . . . >   
'32982','14233','56444','79278','57791',
. . . . . . . . . . . . >   
'37395','93812','14062','21556','58923',
. . . . . . . . . . . . >   
'13595','87261','79484','24492','10389',
. . . . . . . . . . . . >   
'89526','21733','85078','35187','68025',
. . . . . . . . . . . 

[jira] [Created] (DRILL-4093) Use CalciteSchema from Calcite master branch

[Jinfeng Ni (JIRA)]

Jinfeng Ni created DRILL-4093:
-

 Summary: Use CalciteSchema from Calcite master branch
 Key: DRILL-4093
 URL: https://issues.apache.org/jira/browse/DRILL-4093
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Reporter: Jinfeng Ni
Assignee: Jinfeng Ni


Calcite-911 (https://issues.apache.org/jira/browse/CALCITE-911) pushed some 
Drill specific change related to CalciteSchema code to Calcite master branch.  
Drill should pick those changes in the fork, and make adjustment in Drill's 
code.

This would reduce the required effort when Drill wants to rebase the fork onto 
Calcite master, or wants to get rid of the fork.

  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-867) tpcds queries 6, 9 and 10 fail to plan

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007335#comment-15007335
 ] 

Victoria Markman commented on DRILL-867:


I'm going to add these queries to the regression suite as negative tests to 
make sure we don't regress anything. Resolving as fixed for now.

> tpcds queries 6, 9 and 10 fail to plan
> --
>
> Key: DRILL-867
> URL: https://issues.apache.org/jira/browse/DRILL-867
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser
>Reporter: Krystal
>Priority: Minor
> Fix For: Future
>
>
> git.commit.id.abbrev=e1e5ea0
> git.commit.time=29.05.2014 @ 15\:32\:29 PDT
> query 6:
> {code}
> select * from (select  a.ca_state state, count(*) cnt
>  from customer_address a
>  ,customer c
>  ,store_sales s
>  ,date_dim d
>  ,item i
>  where  a.ca_address_sk = c.c_current_addr_sk
>   and c.c_customer_sk = s.ss_customer_sk
>   and s.ss_sold_date_sk = d.d_date_sk
>   and s.ss_item_sk = i.i_item_sk
>   and d.d_month_seq = 
>(select distinct (d.d_month_seq)
> from date_dim d
>where d.d_year = 1998
>   and d.d_moy = 5 ) 
>   and i.i_current_price > 1.2 * 
>  (select avg(j.i_current_price)
>from item j 
>where j.i_category = i.i_category)
>  group by a.ca_state
>  having count(*) >= 10
>  order by cnt
>  ) limit 100;
> {code}
> query 7:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007696#comment-15007696
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/260#issuecomment-157220138
  
How does this address the problem mentioned in the JIRA, which seems to 
come from **StoragePluginMap**?


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3623) Hive query hangs with limit 0 clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007685#comment-15007685
 ] 

ASF GitHub Bot commented on DRILL-3623:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/193#discussion_r45006594
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillDirectScanRel.java
 ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.Iterators;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.physical.DrillScanPrel;
+import org.apache.drill.exec.planner.physical.PhysicalPlanCreator;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.planner.physical.Prel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.store.direct.DirectGroupScan;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Logical and physical RelNode representing a {@link DirectGroupScan}. 
This is not backed by a {@link DrillTable},
+ * unlike {@link DrillScanRel}.
+ */
+public class DrillDirectScanRel extends AbstractRelNode implements 
DrillScanPrel, DrillRel {
--- End diff --

With DrillDirectScanRel, the execution plan is: *Screen <-- Project <-- 
DrillDirectScanRel*. In my experiments ([TPCDS 
queries](https://github.com/mapr/drill-test-framework/tree/master/framework/resources/Functional/tpcds/original/hive)),
 the total time did not cross 0.6 s.

With VALUES operator, there needs to be a LIMIT(0) operator on top. Then, 
the execution plan is: *Screen <-- Project <-- SVR <-- Limit <-- Values*. In my 
experiments, sometimes (seen twice) Project and SVR take ~0.5s to setup, and 
the query take 2s to complete, but this is **not reproducible.**


> Hive query hangs with limit 0 clause
> 
>
> Key: DRILL-3623
> URL: https://issues.apache.org/jira/browse/DRILL-3623
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.1.0
> Environment: MapR cluster
>Reporter: Andries Engelbrecht
>Assignee: Sudheesh Katkam
> Fix For: Future
>
>
> Running a select * from hive.table limit 0 does not return (hangs).
> Select * from hive.table limit 1 works fine
> Hive table is about 6GB with 330 files with parquet using snappy compression.
> Data types are int, bigint, string and double.
> Querying directory with parquet files through the DFS plugin works fine
> select * from dfs.root.`/user/hive/warehouse/database/table` limit 0;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3623) Hive query hangs with limit 0 clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007723#comment-15007723
 ] 

ASF GitHub Bot commented on DRILL-3623:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/193#discussion_r45008717
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillDirectScanRel.java
 ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.Iterators;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.physical.DrillScanPrel;
+import org.apache.drill.exec.planner.physical.PhysicalPlanCreator;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.planner.physical.Prel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.store.direct.DirectGroupScan;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Logical and physical RelNode representing a {@link DirectGroupScan}. 
This is not backed by a {@link DrillTable},
+ * unlike {@link DrillScanRel}.
+ */
+public class DrillDirectScanRel extends AbstractRelNode implements 
DrillScanPrel, DrillRel {
--- End diff --

Should I stick to the current impl?


> Hive query hangs with limit 0 clause
> 
>
> Key: DRILL-3623
> URL: https://issues.apache.org/jira/browse/DRILL-3623
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.1.0
> Environment: MapR cluster
>Reporter: Andries Engelbrecht
>Assignee: Sudheesh Katkam
> Fix For: Future
>
>
> Running a select * from hive.table limit 0 does not return (hangs).
> Select * from hive.table limit 1 works fine
> Hive table is about 6GB with 330 files with parquet using snappy compression.
> Data types are int, bigint, string and double.
> Querying directory with parquet files through the DFS plugin works fine
> select * from dfs.root.`/user/hive/warehouse/database/table` limit 0;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008000#comment-15008000
 ] 

ASF GitHub Bot commented on DRILL-3765:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/255


> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni resolved DRILL-3765.
---
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed in commit: e0c9b841d4001c2574cf015a91974c536528d345

> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.4.0
>
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007690#comment-15007690
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/260#discussion_r45006881
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/zk/ZkAbstractStore.java
 ---
@@ -23,30 +23,35 @@
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map.Entry;
+import java.util.concurrent.TimeUnit;
 
 import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.api.ACLBackgroundPathAndBytesable;
 import org.apache.curator.framework.recipes.cache.ChildData;
 import org.apache.curator.framework.recipes.cache.PathChildrenCache;
 import 
org.apache.curator.framework.recipes.cache.PathChildrenCache.StartMode;
-import org.apache.drill.exec.rpc.data.DataTunnel;
+import org.apache.curator.framework.recipes.locks.InterProcessLock;
--- End diff --

unused imports.


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007692#comment-15007692
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/260#discussion_r45006886
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/zk/ZkAbstractStore.java
 ---
@@ -54,31 +59,26 @@ public ZkAbstractStore(CuratorFramework framework, 
PStoreConfig config)
 this.prefix = parent + "/";
 this.framework = framework;
 this.config = config;
+this.childrenCache = new PathChildrenCache(framework, parent, true);
--- End diff --

unnecessary "this"


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007691#comment-15007691
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/260#discussion_r45006882
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/zk/ZkAbstractStore.java
 ---
@@ -23,30 +23,35 @@
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map.Entry;
+import java.util.concurrent.TimeUnit;
 
 import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.api.ACLBackgroundPathAndBytesable;
 import org.apache.curator.framework.recipes.cache.ChildData;
 import org.apache.curator.framework.recipes.cache.PathChildrenCache;
 import 
org.apache.curator.framework.recipes.cache.PathChildrenCache.StartMode;
-import org.apache.drill.exec.rpc.data.DataTunnel;
+import org.apache.curator.framework.recipes.locks.InterProcessLock;
 import org.apache.drill.exec.store.sys.PStoreConfig;
 import org.apache.zookeeper.CreateMode;
 
 import com.google.common.base.Preconditions;
 import com.google.common.collect.Lists;
+import org.apache.zookeeper.KeeperException;
 
 /**
  * This is the abstract class that is shared by ZkPStore (Persistent 
store) and ZkEStore (Ephemeral Store)
  * @param 
  */
 public abstract class ZkAbstractStore implements AutoCloseable {
-  static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(ZkAbstractStore.class);
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(ZkAbstractStore.class);
 
-  protected CuratorFramework framework;
-  protected PStoreConfig config;
+  protected static final long LOCK_TIMEOUT_MILLIS = 3000;
--- End diff --

unused.


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007693#comment-15007693
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/260#discussion_r45006978
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/zk/ZkAbstractStore.java
 ---
@@ -136,7 +138,81 @@ public boolean putIfAbsent(String key, V value) {
 }
   }
 
-  public abstract void createNodeInZK (String key, V value);
+  /**
+   * Default {@link CreateMode create mode} that will be used in create 
operations referred in the see also section.
+   *
+   * @see #createOrUpdate(String, Object)
+   * @see #createWithPrefix(String, Object)
+   */
+  protected abstract CreateMode getCreateMode();
+
+
+  /**
+   * Creates a node in zookeeper with the {@link #getCreateMode() default 
create mode} and sets its value if supplied.
+   *
+   * @param pathtarget path
+   * @param value   value to set, null if none available
+   *
+   * @see #getCreateMode()
+   * @see #createOrUpdate(String, Object)
+   * @see #withPrefix(String)
+   */
+  protected void createWithPrefix(String path, V value) {
+createOrUpdate(withPrefix(path), value);
+  }
+
+  /**
+   * Creates a node in zookeeper with the {@link #getCreateMode() default 
create mode} and sets its value if supplied
+   * or updates its value if the node already exists.
+   *
+   * Note that if node exists, its mode will not be changed.
+   *
+   * @param pathtarget path
+   * @param value   value to set, null if none available
+   *
+   * @see #getCreateMode()
+   * @see #createOrUpdate(String, Object, CreateMode)
+   */
+  protected void createOrUpdate(String path, V value) {
+createOrUpdate(path, value, getCreateMode());
+  }
+
+  /**
+   * Creates a node in zookeeper with the given mode and sets its value if 
supplied or updates its value if the node
+   * already exists.
+   *
+   * Note that if the node exists, its mode will not be changed.
+   *
+   * Internally, the method suppresses {@link 
org.apache.zookeeper.KeeperException.NodeExistsException}. It is
+   * safe to do so since the implementation is idempotent.
+   *
+   * @param pathtarget path
+   * @param value   value to set, null if none available
+   * @param modecreation mode
+   * @throws RuntimeException  throws a {@link RuntimeException} wrapping 
the root cause.
+   */
+  protected void createOrUpdate(String path, V value, CreateMode mode) {
+try {
+  final boolean isUpdate = value != null;
+  final byte[] valueInBytes = isUpdate ? 
config.getSerializer().serialize(value) : null;
+  final boolean nodeExists = framework.checkExists().forPath(path) != 
null;
+  if (!nodeExists) {
+final ACLBackgroundPathAndBytesable creator = 
framework.create().withMode(mode);
+if (isUpdate) {
+  creator.forPath(path, valueInBytes);
+} else {
+  creator.forPath(path);
+}
+  } else if (isUpdate) {
+framework.setData().forPath(path, valueInBytes);
+  }
+} catch (KeeperException.NodeExistsException ex) {
+  logger.warn("Node already exists in Zookeeper. Skipping... -- [path: 
{}, mode: {}]", path, mode);
--- End diff --

If isUpdate == true, should we try to update?


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3171) Storage Plugins : Two processes tried to update the storage plugin at the same time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007694#comment-15007694
 ] 

ASF GitHub Bot commented on DRILL-3171:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/260#discussion_r45006998
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/zk/ZkPStore.java 
---
@@ -18,8 +18,11 @@
 package org.apache.drill.exec.store.sys.zk;
 
 import java.io.IOException;
+import java.util.concurrent.TimeUnit;
--- End diff --

unused imports.


> Storage Plugins : Two processes tried to update the storage plugin at the 
> same time
> ---
>
> Key: DRILL-3171
> URL: https://issues.apache.org/jira/browse/DRILL-3171
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.0.0
>Reporter: Rahul Challapalli
>Assignee: Deneche A. Hakim
>  Labels: test
> Fix For: Future
>
>
> Commit Id# : bd8ac4fca03ad5043bca27fbc7e0dec5a35ac474
> We have seen this issue happen with the below steps
>1. Clear out the zookeeper
>2. Update the storage plugin using the rest API on one of the node
>3. Submit 10 queries concurrently
> With randomized foreman node selection, the node executing the query might 
> not have the updated storage plugins info. This could be causing the issue.
> - Rahul



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-4035) NPE seen on Functional test run using JDK 8

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007791#comment-15007791
 ] 

Khurram Faraaz edited comment on DRILL-4035 at 11/17/15 1:42 AM:
-

This can also be repro'd on Drill 1.3 on a 4 node cluster (by running 
Functional test suite) that uses Java 1.7 so this is not specific to JDK8. 
NOTE : Assertions were enabled for the functional test run.
git.commit.id=a639c51c

{code}
[root@centos-01 data]# java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

[root@centos-01 data]# javac -version
javac 1.7.0_45

[root@centos-01 drill-1.4.0]# hive --version
Hive 1.0.0-mapr
Subversion git://8116688c8867/root/opensource/mapr-hive-1.0/dl/mapr-hive-1.0 -r 
0d6eee94c9c9106207d3a509f4e6e829320f85cd
Compiled by root on Mon Oct 5 22:13:47 UTC 2015
>From source with checksum da758af192bdbe2079abc9d7c86b4f1a

[root@centos-01 data]# uname -a
Linux centos-0ab 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux
{code}

Here are details from Functional run on Java 1.7
{code}
Failing query => select count(*) from 
hive.dynamic_partitions.lineitem_parquet_partitioned_hive_hier_intint;
in 
Functional/partition_pruning/hive/parquet/dynamic_hier_intint/data/parquetCount1.q

>From output file in drill-output directory

dynamicPartitionDirectoryHive-HierarchicalPartitionDataIntInt_parquetCount1.output_Tue_Nov_17_00:54:12_UTC_2015

0   SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]

>From drillbit.log

2015-11-17 00:54:15,867 [29b5854a-86c3-f778-9fc9-ae5dc4d0d4c9:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
java.lang.NullPointerException
at 
org.apache.drill.common.exceptions.ExecutionSetupException.fromThrowable(ExecutionSetupException.java:30)
 ~[drill-logical-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveRecordReader.setup(HiveRecordReader.java:249)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:108) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:136) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:75)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:36)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
  

[jira] [Created] (DRILL-4096) Incorrect result when we use coalesce in a join condition along with other filters

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4096:


 Summary: Incorrect result when we use coalesce in a join condition 
along with other filters
 Key: DRILL-4096
 URL: https://issues.apache.org/jira/browse/DRILL-4096
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill, Storage - Hive
Affects Versions: 1.3.0
Reporter: Rahul Challapalli
Priority: Critical


git.commit.id.abbrev=447f8ba

The below query returns no results which is wrong based on the data set. 
Interestingly if we remove the second filter we get a cannot plan exception 
from drill. Will raise a different jira, if I cannot find an existing one
{code}
select * from hive.null_schemachange d, hive.onlynulls n where d.date_col = 
coalesce(n.date_col, date '2038-04-10', n.date_col) and d.date_col > 
'2015-01-01';
{code}

Hive DDL :
{code}
drop table if exists null_schemachange;
create external table null_schemachange (
  int_col int,
  bigint_col bigint,
  date_col date,
  time_col string,
  timestamp_col timestamp,
  interval_col string,
  varchar_col string,
  float_col float,
  double_col double,
  bool_col boolean
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY "|"
LOCATION '/drill/testdata/hive_storage/null_schemachange.tbl'
TBLPROPERTIES ("serialization.null.format"="null");


drop table if exists onlynulls;
create external table onlynulls (
  int_col int,
  bigint_col bigint,
  date_col date,
  time_col string,
  timestamp_col timestamp,
  interval_col string,
  varchar_col string,
  float_col float,
  double_col double,
  bool_col boolean
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY "|"
LOCATION '/drill/testdata/hive_storage/onlynulls.tbl'
TBLPROPERTIES ("serialization.null.format"="null");
{code}

The data files are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4096) Incorrect result when we use coalesce in a join condition along with other filters

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007802#comment-15007802
 ] 

Rahul Challapalli commented on DRILL-4096:
--

The below file has schema changes which hive treats as null based on the 
defined type
null_schema.tbl
{code}
1|98723980547|null|00:00:00|null|P18582D|jllkjsdhfg|2345.33|null|false
null|24509823475|null|01:00:00|1997-01-02 
00:00:00.0|P1DT9045S|jhgduitweriuoert|3243.32|664522.332|true
3|05:10:36|null|null|null|P1DT9000S|gfdstweopiu|234533.2|234522.3221|true
4|null|1996-01-29|true|null|P10DT9000S|gjklhsdfgkjhkASDF|null|234543245.44|false
5|39575748|1996-03-01|12:00:00|P1DT9045|P1DT9045.100S|oieoiutriotureWERTgwgEWRg|23.5|654.4|false
6|47509823|1996-03-02|12:01:00|1997-02-11 
17:32:01.0|-23|gjkdfkjglfd|2345552345.5342|null|false
7|7520398475|1997-02-28|23:59:00|1997-02-12 
17:32:01.0|P2Y|-0|211.2|798276340.9|true
8|null|null|23:59:59.990|1997-02-13 
17:32:01.0|null|lkjgfiurtoUYFHfahui|9823.435|98723.879|false
9|509238475|1997-03-01|15:36:39|null|P1DT9000S|IOUfiuodsfIUfjkh|null|98273.455|true
a|4345234|1997-03-02|15:36:39|1997-02-15 
17:32:01.0|P10DT9000S|iweuoHUIhUwer|425.43|4756389.345|false
null|980347589|2000-04-01|00:01:00|1997-02-16 
17:32:01.0|P1DT9045.100S|KfijUIwre|3433.3|87364.5|true
12|859702345|2000-04-03|null|1897-02-18 
17:32:01.0|P1DT9045S|dfgoiuert|34980.43|78346.3452|false
13|57784356923|2038-04-08|23:59:00|2002-02-14 
17:32:01.0|P2Y|uitreo|3425.543|765234.43|null
14|302|2039-04-09|12:03:00|1991-02-10 
17:32:01.0|P1Y2M|uigoMnvjjkdf|3654.43|763478.5623|false
15|2983475089|2040-04-10|null|1900-02-16 
17:32:01.0|P10DT9000S|NvvdfHVG|1997-02-11 17:32:01.0|8767985.43|true
16|null|null|19:59:00|null|P18582D|null|4784.5735|2037598.58734|null
null|3750982435|1999-02-08|11:59:59.990|1897-02-16 
17:32:01.0|P1DT9045S|Nhkhuivb|6459.592|6538.7832|kjsahfd
18|4582345|1999-03-08|null|1997-02-16 
17:32:01.0|null|uiuikjk|null|68791189.43|null
null|982|1999-01-18|15:36:39|null|P1DT9045.100S|null|645.2321|87623.34|true
20|7893759083|dsa|11:59:59.990|1996-02-28 
17:32:01.0|P1DT9000S|hjiwgh|345.543|false|false
null|null|null|null|null|null|null|null|null|null
|
{code}

onlynulls.tbl :
{code}
null|null|null|null|null|null|null|null|null|null
|
{code}

> Incorrect result when we use coalesce in a join condition along with other 
> filters
> --
>
> Key: DRILL-4096
> URL: https://issues.apache.org/jira/browse/DRILL-4096
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill, Storage - Hive
>Affects Versions: 1.3.0
>Reporter: Rahul Challapalli
>Priority: Critical
>
> git.commit.id.abbrev=447f8ba
> The below query returns no results which is wrong based on the data set. 
> Interestingly if we remove the second filter we get a cannot plan exception 
> from drill. Will raise a different jira, if I cannot find an existing one
> {code}
> select * from hive.null_schemachange d, hive.onlynulls n where d.date_col = 
> coalesce(n.date_col, date '2038-04-10', n.date_col) and d.date_col > 
> '2015-01-01';
> {code}
> Hive DDL :
> {code}
> drop table if exists null_schemachange;
> create external table null_schemachange (
>   int_col int,
>   bigint_col bigint,
>   date_col date,
>   time_col string,
>   timestamp_col timestamp,
>   interval_col string,
>   varchar_col string,
>   float_col float,
>   double_col double,
>   bool_col boolean
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY "|"
> LOCATION '/drill/testdata/hive_storage/null_schemachange.tbl'
> TBLPROPERTIES ("serialization.null.format"="null");
> drop table if exists onlynulls;
> create external table onlynulls (
>   int_col int,
>   bigint_col bigint,
>   date_col date,
>   time_col string,
>   timestamp_col timestamp,
>   interval_col string,
>   varchar_col string,
>   float_col float,
>   double_col double,
>   bool_col boolean
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY "|"
> LOCATION '/drill/testdata/hive_storage/onlynulls.tbl'
> TBLPROPERTIES ("serialization.null.format"="null");
> {code}
> The data files are attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4097) AssertionError: Internal error: Error while applying rule DrillTableRule

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4097:
-

 Summary: AssertionError: Internal error: Error while applying rule 
DrillTableRule
 Key: DRILL-4097
 URL: https://issues.apache.org/jira/browse/DRILL-4097
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.3.0
Reporter: Khurram Faraaz


Test fails when assertions are enabled due to below assertion error.

Caused by: java.lang.AssertionError: Internal error: Error while applying rule 
DrillTableRule, args 
[rel#502692:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, 
/drill/testdata/metadata_caching/lineitem_deletecache])]

4 node cluster on CentOS, Drill 1.3, git.commit.id=a639c51c

Failing query is from 
resources/Functional/metadata_caching/data/mc1_deletetoplevelcache1.q

{code}
select count(*) from 
dfs.`/drill/testdata/metadata_caching/lineitem_deletecache` where dir0=2006;
{code}

Stack trace from drillbit.log

{code}
2015-11-17 00:28:52,655 [29b58b3b-0142-5bcc-aaa0-4458784e61ee:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: JsonMappingException: No 
content to map due to end-of-input
 at [Source: com.mapr.fs.MapRFsDataInputStream@68ec4bc6; line: 1, column: 1]


[Error Id: f1c7ae51-eb7b-4eb1-9c90-6e16a6905b95 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
JsonMappingException: No content to map due to end-of-input
 at [Source: com.mapr.fs.MapRFsDataInputStream@68ec4bc6; line: 1, column: 1]


[Error Id: f1c7ae51-eb7b-4eb1-9c90-6e16a6905b95 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Internal error: Error while applying 
rule DrillTableRule, args 
[rel#502692:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, 
/drill/testdata/metadata_caching/lineitem_deletecache])]
... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: Error while applying rule 
DrillTableRule, args 
[rel#502692:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, 
/drill/testdata/metadata_caching/lineitem_deletecache])]
at org.apache.calcite.util.Util.newInternal(Util.java:792) 
~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
 ~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
 ~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
at 
org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303) 
~[calcite-core-1.4.0-drill-r8.jar:1.4.0-drill-r8]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
 

[jira] [Commented] (DRILL-3623) Hive query hangs with limit 0 clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007751#comment-15007751
 ] 

ASF GitHub Bot commented on DRILL-3623:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/193#discussion_r45010657
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillDirectScanRel.java
 ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.Iterators;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.physical.DrillScanPrel;
+import org.apache.drill.exec.planner.physical.PhysicalPlanCreator;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.planner.physical.Prel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.store.direct.DirectGroupScan;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Logical and physical RelNode representing a {@link DirectGroupScan}. 
This is not backed by a {@link DrillTable},
+ * unlike {@link DrillScanRel}.
+ */
+public class DrillDirectScanRel extends AbstractRelNode implements 
DrillScanPrel, DrillRel {
--- End diff --

Sounds like that the DrillDirectScanRel approach works only when "limit 0" 
is on the root level; it would not apply when "limit 0" is inside a subquery. 

The Values approach may apply for the case "limit 0" in a subquery. 
Certainly, this JIRA only targets the case of "limit 0" on root. But with the 
VALUES approach, there is room for improvement for a more general scenarios. 




> Hive query hangs with limit 0 clause
> 
>
> Key: DRILL-3623
> URL: https://issues.apache.org/jira/browse/DRILL-3623
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.1.0
> Environment: MapR cluster
>Reporter: Andries Engelbrecht
>Assignee: Sudheesh Katkam
> Fix For: Future
>
>
> Running a select * from hive.table limit 0 does not return (hangs).
> Select * from hive.table limit 1 works fine
> Hive table is about 6GB with 330 files with parquet using snappy compression.
> Data types are int, bigint, string and double.
> Querying directory with parquet files through the DFS plugin works fine
> select * from dfs.root.`/user/hive/warehouse/database/table` limit 0;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3765) Partition prune rule is unnecessary fired multiple times.

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007761#comment-15007761
 ] 

Jinfeng Ni commented on DRILL-3765:
---

I should point out that the improvement from this patch depends on how much 
time is spent on evaluation of pruning filter in the rule in the overall 
planning time.  The more complex pruning filter we have, the more likely we 
would see improvements.

For example, in the previous comment, I compared the planning time for a 
pruning filter with a 5 value in-list, (5.2 seconds vs 9.4 seconds).  If the 
pruning filter is changed to one = condition, then the planning time is changed 
to 3.7 seconds vs 4.6 seconds. That is, we would see smaller improvement with 
simpler pruning filter, which seems to be reasonable. 

{code}
explain plan for select ss_sold_date_sk, ss_sold_time_sk, ss_item_sk, 
ss_customer_sk from dfs.tmp.store_pb_item_sk where ss_item_sk =100  and 
ss_customer_sk = 96479;

1 row selected (3.709 seconds)

alter session set `planner.enable_hep_partition_pruning` = false;

1 row selected (4.65 seconds)
{code}


> Partition prune rule is unnecessary fired multiple times. 
> --
>
> Key: DRILL-3765
> URL: https://issues.apache.org/jira/browse/DRILL-3765
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
>
> It seems that the partition prune rule may be fired multiple times, even 
> after the first rule execution has pushed the filter into the scan operator. 
> Since partition prune has to build the vectors to contain the partition /file 
> / directory information, to invoke the partition prune rule unnecessary may 
> lead to big memory overhead.
> Drill planner should avoid the un-necessary partition prune rule, in order to 
> reduce the chance of hitting OOM exception, while the partition prune rule is 
> executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3623) Hive query hangs with limit 0 clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007785#comment-15007785
 ] 

ASF GitHub Bot commented on DRILL-3623:
---

Github user julianhyde commented on a diff in the pull request:

https://github.com/apache/drill/pull/193#discussion_r45012029
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillDirectScanRel.java
 ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.logical;
+
+import com.google.common.collect.Iterators;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.RelWriter;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.drill.common.logical.data.LogicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.planner.physical.DrillScanPrel;
+import org.apache.drill.exec.planner.physical.PhysicalPlanCreator;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.planner.physical.Prel;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+import org.apache.drill.exec.planner.physical.visitor.PrelVisitor;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.store.direct.DirectGroupScan;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Logical and physical RelNode representing a {@link DirectGroupScan}. 
This is not backed by a {@link DrillTable},
+ * unlike {@link DrillScanRel}.
+ */
+public class DrillDirectScanRel extends AbstractRelNode implements 
DrillScanPrel, DrillRel {
--- End diff --

Yeah, I think DrillDirectScanRel is a wrong turn for this functionality, 
because you can't see that it is empty and reason about it. A DirectGroupScan 
is a runtime thing, so shouldn't be floating around at planning time.

I notice that DrillValuesRel does not extend Values. If it did, it would 
get all of the rules in PruneEmptyRules for free - pruning away Filter, Project 
etc. on top of it.


> Hive query hangs with limit 0 clause
> 
>
> Key: DRILL-3623
> URL: https://issues.apache.org/jira/browse/DRILL-3623
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.1.0
> Environment: MapR cluster
>Reporter: Andries Engelbrecht
>Assignee: Sudheesh Katkam
> Fix For: Future
>
>
> Running a select * from hive.table limit 0 does not return (hangs).
> Select * from hive.table limit 1 works fine
> Hive table is about 6GB with 330 files with parquet using snappy compression.
> Data types are int, bigint, string and double.
> Querying directory with parquet files through the DFS plugin works fine
> select * from dfs.root.`/user/hive/warehouse/database/table` limit 0;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4035) NPE seen on Functional test run using JDK 8

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007791#comment-15007791
 ] 

Khurram Faraaz commented on DRILL-4035:
---

I can also repro this on Drill 1.3 on a 4 node cluster (by running Functional 
test suite) that uses Java 1.7 so this is not specific to JDK8.
git.commit.id=a639c51c

{code}
[root@centos-01 data]# java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

[root@centos-01 data]# javac -version
javac 1.7.0_45

[root@centos-01 data]# uname -a
Linux centos-0ab 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux
{code}

Here are details from Functional run on Java 1.7
{code}
Failing query => select count(*) from 
hive.dynamic_partitions.lineitem_parquet_partitioned_hive_hier_intint;
in 
Functional/partition_pruning/hive/parquet/dynamic_hier_intint/data/parquetCount1.q

>From output file in drill-output directory

dynamicPartitionDirectoryHive-HierarchicalPartitionDataIntInt_parquetCount1.output_Tue_Nov_17_00:54:12_UTC_2015

0   SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]

>From drillbit.log

2015-11-17 00:54:15,867 [29b5854a-86c3-f778-9fc9-ae5dc4d0d4c9:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
java.lang.NullPointerException
at 
org.apache.drill.common.exceptions.ExecutionSetupException.fromThrowable(ExecutionSetupException.java:30)
 ~[drill-logical-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveRecordReader.setup(HiveRecordReader.java:249)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:108) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:136) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:75)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:36)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 

[jira] [Comment Edited] (DRILL-4035) NPE seen on Functional test run using JDK 8

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007791#comment-15007791
 ] 

Khurram Faraaz edited comment on DRILL-4035 at 11/17/15 1:40 AM:
-

I can also repro this on Drill 1.3 on a 4 node cluster (by running Functional 
test suite) that uses Java 1.7 so this is not specific to JDK8.
git.commit.id=a639c51c

{code}
[root@centos-01 data]# java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

[root@centos-01 data]# javac -version
javac 1.7.0_45

[root@centos-01 drill-1.4.0]# hive --version
Hive 1.0.0-mapr
Subversion git://8116688c8867/root/opensource/mapr-hive-1.0/dl/mapr-hive-1.0 -r 
0d6eee94c9c9106207d3a509f4e6e829320f85cd
Compiled by root on Mon Oct 5 22:13:47 UTC 2015
>From source with checksum da758af192bdbe2079abc9d7c86b4f1a

[root@centos-01 data]# uname -a
Linux centos-0ab 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux
{code}

Here are details from Functional run on Java 1.7
{code}
Failing query => select count(*) from 
hive.dynamic_partitions.lineitem_parquet_partitioned_hive_hier_intint;
in 
Functional/partition_pruning/hive/parquet/dynamic_hier_intint/data/parquetCount1.q

>From output file in drill-output directory

dynamicPartitionDirectoryHive-HierarchicalPartitionDataIntInt_parquetCount1.output_Tue_Nov_17_00:54:12_UTC_2015

0   SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]

>From drillbit.log

2015-11-17 00:54:15,867 [29b5854a-86c3-f778-9fc9-ae5dc4d0d4c9:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
NullPointerException

Fragment 0:0

[Error Id: 299d8478-69a2-49ec-96f8-b7bd18261d35 on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
java.lang.NullPointerException
at 
org.apache.drill.common.exceptions.ExecutionSetupException.fromThrowable(ExecutionSetupException.java:30)
 ~[drill-logical-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveRecordReader.setup(HiveRecordReader.java:249)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:108) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:136) 
~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:75)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:36)
 ~[drill-storage-hive-core-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
 ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 

[jira] [Commented] (DRILL-3827) Empty metadata file causes queries on the table to fail

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007816#comment-15007816
 ] 

Khurram Faraaz commented on DRILL-3827:
---


{code}

Note that file is owned by user mapr.

[root@centos-01 ~]# hadoop fs -ls /tmp/CTAS_ONE_MILN_RWS_PER_GROUP
Found 99 items
-rwxr-xr-x   3 mapr mapr  68554 2015-11-17 01:44 
/tmp/CTAS_ONE_MILN_RWS_PER_GROUP/.drill.parquet_metadata

Metadata cache file is removed.

root@centos-01 ~]# hadoop fs -rmr 
/tmp/CTAS_ONE_MILN_RWS_PER_GROUP/.drill.parquet_metadata
rmr: DEPRECATED: Please use 'rm -r' instead.
15/11/17 01:47:01 INFO Configuration.deprecation: io.bytes.per.checksum is 
deprecated. Instead, use dfs.bytes-per-checksum
15/11/17 01:47:01 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /tmp/CTAS_ONE_MILN_RWS_PER_GROUP/.drill.parquet_metadata

replace with an empty file
[root@centos-01 ~]# hadoop fs -put .drill.parquet_metadata 
/tmp/CTAS_ONE_MILN_RWS_PER_GROUP/

Note that file is now owned by root
[root@centos-01 ~]# hadoop fs -ls /tmp/CTAS_ONE_MILN_RWS_PER_GROUP
Found 99 items
-rwxr-xr-x   3 root root  0 2015-11-17 01:48 
/tmp/CTAS_ONE_MILN_RWS_PER_GROUP/.drill.parquet_metadata
{code}

{code}
[root@centos-01 bin]# ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
apache drill 1.4.0-SNAPSHOT
"got drill?"
0: jdbc:drill:schema=dfs.tmp> select count(*) from CTAS_ONE_MILN_RWS_PER_GROUP;
+---+
|  EXPR$0   |
+---+
| 5000  |
+---+
1 row selected (0.908 seconds)
0: jdbc:drill:schema=dfs.tmp> refresh table metadata 
CTAS_ONE_MILN_RWS_PER_GROUP;
+---+---+
|  ok   |summary
|
+---+---+
| true  | Successfully updated metadata for table CTAS_ONE_MILN_RWS_PER_GROUP.  
|
+---+---+
1 row selected (0.607 seconds)
0: jdbc:drill:schema=dfs.tmp> select count(*) from CTAS_ONE_MILN_RWS_PER_GROUP;
+---+
|  EXPR$0   |
+---+
| 5000  |
+---+
1 row selected (0.439 seconds)
...
0: jdbc:drill:schema=dfs.tmp> select count(*) from CTAS_ONE_MILN_RWS_PER_GROUP;
Error: SYSTEM ERROR: JsonMappingException: No content to map due to end-of-input
 at [Source: com.mapr.fs.MapRFsDataInputStream@5bf94307; line: 1, column: 1]


[Error Id: 5826fd6a-c99e-4cb0-940f-8180a663932b on centos-01.qa.lab:31010] 
(state=,code=0)
{code}

Here is the stack trace from drillbit.log

{code}
2015-11-17 01:48:34,519 [29b5788c-b51c-bba9-3328-998e82a3b462:foreman] ERROR 
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: JsonMappingException: No 
content to map due to end-of-input
 at [Source: com.mapr.fs.MapRFsDataInputStream@5bf94307; line: 1, column: 1]


[Error Id: 5826fd6a-c99e-4cb0-940f-8180a663932b on centos-01.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
JsonMappingException: No content to map due to end-of-input
 at [Source: com.mapr.fs.MapRFsDataInputStream@5bf94307; line: 1, column: 1]


[Error Id: 5826fd6a-c99e-4cb0-940f-8180a663932b on centos-01.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
 [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
exception during fragment initialization: Internal error: Error while applying 
rule DrillPushProjIntoScan, args 

[jira] [Commented] (DRILL-3854) IOB Exception : CONVERT_FROM (sal, int_be)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008132#comment-15008132
 ] 

ASF GitHub Bot commented on DRILL-3854:
---

GitHub user hsuanyi opened a pull request:

https://github.com/apache/drill/pull/262

DRILL-3854: For convert_from, re-point readerIndex in DrillBuf at the…

… beginning after evaluation is finished

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hsuanyi/incubator-drill DRILL-3854

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit dddae2686262e834739d31bf2296a82e0616aeee
Author: Hsuan-Yi Chu 
Date:   2015-11-14T23:34:56Z

DRILL-3854: For convert_from, re-point readerIndex in DrillBuf at the 
beginning after evaluation is finished




> IOB Exception : CONVERT_FROM (sal, int_be)
> --
>
> Key: DRILL-3854
> URL: https://issues.apache.org/jira/browse/DRILL-3854
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: log, run_time_code.txt
>
>
> CONVERT_FROM function results in IOB Exception
> Drill master commit id : b9afcf8f
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select salary from Emp;
> +-+
> | salary  |
> +-+
> | 8   |
> | 9   |
> | 20  |
> | 95000   |
> | 85000   |
> | 9   |
> | 10  |
> | 87000   |
> | 8   |
> | 10  |
> | 99000   |
> +-+
> 11 rows selected (0.535 seconds)
> # create table using above Emp table
> create table tbl_int_be as select convert_to(salary, 'int_be') sal from Emp;
> 0: jdbc:drill:schema=dfs.tmp> alter session set `planner.slice_target`=1;
> +---++
> |  ok   |summary |
> +---++
> | true  | planner.slice_target updated.  |
> +---++
> 1 row selected (0.19 seconds)
> # Below query results in IOB on server.
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be') from 
> tbl_int_be order by sal;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: DrillBuf(ridx: 0, widx: 158, 
> cap: 158/158, unwrapped: SlicedByteBuf(ridx: 0, widx: 158, cap: 158/158, 
> unwrapped: UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 
> 0, cap: 417/417.slice(158, 44)
> Fragment 2:0
> [Error Id: 4ee1361d-9877-45eb-bde6-57d5add9fe5e on centos-04.qa.lab:31010] 
> (state=,code=0)
> # Apply convert_from function and project original column results in IOB on 
> client. (because Error Id is missing)
> 0: jdbc:drill:schema=dfs.tmp> select convert_from(sal, 'int_be'), sal from 
> tbl_int_be;
> Error: Unexpected RuntimeException: java.lang.IndexOutOfBoundsException: 
> DrillBuf(ridx: 0, widx: 114, cap: 114/114, unwrapped: DrillBuf(ridx: 321, 
> widx: 321, cap: 321/321, unwrapped: 
> UnsafeDirectLittleEndian(PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 
> 321/321.slice(55, 103) (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4098) Assembly code in drillbit.log

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4098:
-

 Summary: Assembly code in drillbit.log
 Key: DRILL-4098
 URL: https://issues.apache.org/jira/browse/DRILL-4098
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.3.0
Reporter: Khurram Faraaz


We are seeing the below assembly code and the stack trace in drillbit.log after 
a Functional test run on a 4 node cluster on CentOS using MapR-Drill 1.3 RPM 
(latest 11/11).

>From drillbit.log
{code}
2015-11-12 06:36:53,553 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 29bbcc7a-36dd-dc7a-d77a-388b228896a4:0:0: 
State to report: RUNNING
2015-11-12 06:36:53,588 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] ERROR 
o.a.drill.exec.compile.MergeAdapter - Failure while merging classes.
java.lang.RuntimeException: Error at instruction 26: Expected an object 
reference, but found . doEval(II)V
0 R I I . . .  :  :L0
1 R I I . . .  :  : LINENUMBER 104 L0
2 R I I . . .  :  : LCONST_0
3 R I I . . .  : J  : LSTORE 3
4 R I I J . .  :  :L1
5 R I I J . .  :  : LINENUMBER 106 L1
6 R I I J . .  :  : ALOAD 0
7 R I I J . .  : R  : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv0 : 
Lorg/apache/drill/exec/vector/BigIntVector;
8 R I I J . .  : R  : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getAccessor 
()Lorg/apache/drill/exec/vector/BigIntVector$Accessor;
9 R I I J . .  : R  : ILOAD 1
00010 R I I J . .  : R I  : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Accessor.get (I)J
00011 R I I J . .  : J  : LSTORE 3
00012 R I I J . .  :  :L2
00013 R I I J . .  :  : LINENUMBER 108 L2
00014 R I I J . .  :  : ALOAD 0
00015 R I I J . .  : R  : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv4 : 
Lorg/apache/drill/exec/vector/BigIntVector;
00016 R I I J . .  : R  : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getMutator 
()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
00017 R I I J . .  : R  : ILOAD 2
00018 R I I J . .  : R I  : LLOAD 3
00019 R I I J . .  : R I J  : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
00020 R I I J . .  :  :L3
00021 R I I J . .  :  : LINENUMBER 109 L3
00022 R I I J . .  :  : ALOAD 0
00023 R I I J . .  : R  : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv7 : 
Lorg/apache/drill/exec/vector/BigIntVector;
00024 R I I J . .  : R  : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getMutator 
()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
00025 R I I J . .  : R  : ILOAD 2
00026 R I I J . .  : R I  : ALOAD 5
00027 ?   : GETFIELD 
org/apache/drill/exec/expr/holders/BigIntHolder.value : J
00028 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
00029 ?   :L4
00030 ?   : LINENUMBER 110 L4
00031 ?   : ALOAD 0
00032 ?   : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv10 : 
Lorg/apache/drill/exec/vector/BigIntVector;
00033 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getMutator 
()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
00034 ?   : ILOAD 2
00035 ?   : ALOAD 5
00036 ?   : GETFIELD 
org/apache/drill/exec/expr/holders/BigIntHolder.value : J
00037 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
00038 ?   :L5
00039 ?   : LINENUMBER 111 L5
00040 ?   : ALOAD 0
00041 ?   : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv13 : 
Lorg/apache/drill/exec/vector/BigIntVector;
00042 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getMutator 
()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
00043 ?   : ILOAD 2
00044 ?   : ALOAD 5
00045 ?   : GETFIELD 
org/apache/drill/exec/expr/holders/BigIntHolder.value : J
00046 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
00047 ?   :L6
00048 ?   : LINENUMBER 112 L6
00049 ?   : ALOAD 0
00050 ?   : GETFIELD 
org/apache/drill/exec/test/generated/ProjectorGen4245.vv16 : 
Lorg/apache/drill/exec/vector/BigIntVector;
00051 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector.getMutator 
()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
00052 ?   : ILOAD 2
00053 ?   : ALOAD 5
00054 ?   : GETFIELD 
org/apache/drill/exec/expr/holders/BigIntHolder.value : J
00055 ?   : INVOKEVIRTUAL 
org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
00056 ?   

[jira] [Commented] (DRILL-4013) Umbrella: Add Support for client warnings

[Parth Chandra (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008194#comment-15008194
 ] 

Parth Chandra commented on DRILL-4013:
--

[~apol] I've added some comments in the doc. 

> Umbrella: Add Support for client warnings
> -
>
> Key: DRILL-4013
> URL: https://issues.apache.org/jira/browse/DRILL-4013
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - JDBC, Client - ODBC, Execution - RPC
>Reporter: Jacques Nadeau
>Assignee: Abhijit Pol
> Fix For: 1.4.0
>
>
> We need to add a capability to support JDBC and ODBC warnings. We should come 
> up with a proposal for a design. For v1, I propose that we focus on Screen 
> operator warnings. Future enhancements could leverage this functionality to 
> provide sideband warnings as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4098) Assembly code in drillbit.log

[Sean Hsuan-Yi Chu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008183#comment-15008183
 ] 

Sean Hsuan-Yi Chu commented on DRILL-4098:
--

Drill tries to apply scalar replacement, and if it fails (just like your 
observation, exception comes along with some assembly code), it falls back to 
the original code path. 

So as long as the original code path works, it might be fine. How did the 
result in client look? Was is correct? 

> Assembly code in drillbit.log
> -
>
> Key: DRILL-4098
> URL: https://issues.apache.org/jira/browse/DRILL-4098
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.3.0
>Reporter: Khurram Faraaz
>
> We are seeing the below assembly code and the stack trace in drillbit.log 
> after a Functional test run on a 4 node cluster on CentOS using MapR-Drill 
> 1.3 RPM (latest 11/11).
> From drillbit.log
> {code}
> 2015-11-12 06:36:53,553 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 29bbcc7a-36dd-dc7a-d77a-388b228896a4:0:0: State to report: RUNNING
> 2015-11-12 06:36:53,588 [29bbcc7a-36dd-dc7a-d77a-388b228896a4:frag:0:0] ERROR 
> o.a.drill.exec.compile.MergeAdapter - Failure while merging classes.
> java.lang.RuntimeException: Error at instruction 26: Expected an object 
> reference, but found . doEval(II)V
> 0 R I I . . .  :  :L0
> 1 R I I . . .  :  : LINENUMBER 104 L0
> 2 R I I . . .  :  : LCONST_0
> 3 R I I . . .  : J  : LSTORE 3
> 4 R I I J . .  :  :L1
> 5 R I I J . .  :  : LINENUMBER 106 L1
> 6 R I I J . .  :  : ALOAD 0
> 7 R I I J . .  : R  : GETFIELD 
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv0 : 
> Lorg/apache/drill/exec/vector/BigIntVector;
> 8 R I I J . .  : R  : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector.getAccessor 
> ()Lorg/apache/drill/exec/vector/BigIntVector$Accessor;
> 9 R I I J . .  : R  : ILOAD 1
> 00010 R I I J . .  : R I  : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector$Accessor.get (I)J
> 00011 R I I J . .  : J  : LSTORE 3
> 00012 R I I J . .  :  :L2
> 00013 R I I J . .  :  : LINENUMBER 108 L2
> 00014 R I I J . .  :  : ALOAD 0
> 00015 R I I J . .  : R  : GETFIELD 
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv4 : 
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00016 R I I J . .  : R  : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector.getMutator 
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00017 R I I J . .  : R  : ILOAD 2
> 00018 R I I J . .  : R I  : LLOAD 3
> 00019 R I I J . .  : R I J  : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00020 R I I J . .  :  :L3
> 00021 R I I J . .  :  : LINENUMBER 109 L3
> 00022 R I I J . .  :  : ALOAD 0
> 00023 R I I J . .  : R  : GETFIELD 
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv7 : 
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00024 R I I J . .  : R  : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector.getMutator 
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00025 R I I J . .  : R  : ILOAD 2
> 00026 R I I J . .  : R I  : ALOAD 5
> 00027 ?   : GETFIELD 
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00028 ?   : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00029 ?   :L4
> 00030 ?   : LINENUMBER 110 L4
> 00031 ?   : ALOAD 0
> 00032 ?   : GETFIELD 
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv10 : 
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00033 ?   : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector.getMutator 
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00034 ?   : ILOAD 2
> 00035 ?   : ALOAD 5
> 00036 ?   : GETFIELD 
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00037 ?   : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector$Mutator.set (IJ)V
> 00038 ?   :L5
> 00039 ?   : LINENUMBER 111 L5
> 00040 ?   : ALOAD 0
> 00041 ?   : GETFIELD 
> org/apache/drill/exec/test/generated/ProjectorGen4245.vv13 : 
> Lorg/apache/drill/exec/vector/BigIntVector;
> 00042 ?   : INVOKEVIRTUAL 
> org/apache/drill/exec/vector/BigIntVector.getMutator 
> ()Lorg/apache/drill/exec/vector/BigIntVector$Mutator;
> 00043 ?   : ILOAD 2
> 00044 ?   : ALOAD 5
> 00045 ?   : GETFIELD 
> org/apache/drill/exec/expr/holders/BigIntHolder.value : J
> 00046 ?   : INVOKEVIRTUAL 
>