[jira] [Created] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

2017-02-14 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-5263:


 Summary: Prevent left NLJoin with non scalar subqueries
 Key: DRILL-5263
 URL: https://issues.apache.org/jira/browse/DRILL-5263
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join and non scalar sub-queries. Drill 
should throw error in this case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5237) Same query produces different/wrong results

2017-02-01 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-5237:


 Summary: Same query produces different/wrong results
 Key: DRILL-5237
 URL: https://issues.apache.org/jira/browse/DRILL-5237
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Query 
{code:sql}
select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as b 
from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 'a1');
{code}
returns different results for different values for option 
planner.width.max_per_node
With options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 1;
{code}
query returns correct result:
{noformat}
+--+
| col  |
+--+
| 3|
+--+
{noformat}
but with options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 3;
{code}
the same query returns wrong result 
{noformat}
+--+
| col  |
+--+
| 2|
+--+
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (DRILL-1808) Large compilation unit tests fails due to high memory allocation

2016-12-20 Thread Serhii Harnyk (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-1808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk resolved DRILL-1808.
--
   Resolution: Duplicate
Fix Version/s: (was: Future)
   1.10.0

Fixed in 
[810198b|https://github.com/apache/drill/commit/810198b18bfbe38712256c58b102bca079d934c1]

> Large compilation unit tests fails due to high memory allocation
> 
>
> Key: DRILL-1808
> URL: https://issues.apache.org/jira/browse/DRILL-1808
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Reporter: Steven Phillips
>Priority: Minor
> Fix For: 1.10.0
>
>
> The buildSchema method of external sort calls allocateNew() on each vector in 
> schema batch, which normally isn't much memory, but when there are thousands 
> of columns, this ends up being significant. We should avoid allocating any 
> memory for the schema batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5117) Compile error when query a json file with 1000+columns

2016-12-09 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-5117:


 Summary: Compile error when query a json file with 1000+columns
 Key: DRILL-5117
 URL: https://issues.apache.org/jira/browse/DRILL-5117
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Codegen
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


Query failed with compile error when we querying a json file with 1000+columns:
{noformat}
0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] 
(state=,code=0)
{noformat}

Stack trace from sqlline.log:
{noformat}
2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
27b54af4-b41f-0682-e50d-626de4eff68e: select * from 
dfs.`/tmp/tooManyFields.json` limit 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
localhost.  Skipping affinity to that host.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, Average 
start: 6.956000 μs .
2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State to report: RUNNING
2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN  
o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - 
probably you're running Drill with a JRE and not a JDK
2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested RUNNING --> FAILED
2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested FAILED --> FINISHED
2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: 
Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows 
beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.8.0.jar:1.8.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
 [drill-java-exec-1.8.0.jar:1.8.0]
at

[jira] [Created] (DRILL-5048) AssertionError when case statement is used with timestamp and null

2016-11-17 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-5048:


 Summary: AssertionError when case statement is used with timestamp 
and null
 Key: DRILL-5048
 URL: https://issues.apache.org/jira/browse/DRILL-5048
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


AssertionError when we use case with timestamp and null:

{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
. . . . . . . . . . . . . . > FROM
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
. . . . . . . . . . . . . . > FROM (values(1)) foo
. . . . . . . . . . . . . . > ) foobar;
Error: SYSTEM ERROR: AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL


[Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log

{noformat}
Caused by: java.lang.AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
... 16 common frames omitted
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

2016-11-10 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-5032:


 Summary: Drill query on hive parquet table failed with 
OutOfMemoryError: Java heap space
 Key: DRILL-5032
 URL: https://issues.apache.org/jira/browse/DRILL-5032
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Hive
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Following query on hive parquet table failed with OOM Java heap space:
{code}
select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 1 ms
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 3 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. 
Information message: Unable to handle out of memory condition in Foreman.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) 
~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
 ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:166) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.print(TextFormat.java:71) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.printToString(TextFormat.java:118) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:106) 
~[protobuf-java-2.5.0.jar:na]
at

[jira] [Created] (DRILL-4906) CASE Expression with constant generates class exception

2016-09-27 Thread Serhii Harnyk (JIRA) 
Serhii Harnyk created DRILL-4906:


 Summary: CASE Expression with constant generates class exception
 Key: DRILL-4906
 URL: https://issues.apache.org/jira/browse/DRILL-4906
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser
Affects Versions: 1.8.0, 1.6.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: 1.9.0


How to reproduce:

select (case when (true) then 1 end) from (values(1));

Error
Error: SYSTEM ERROR: ClassCastException: 
org.apache.drill.exec.expr.holders.NullableVarCharHolder cannot be cast to 
org.apache.drill.exec.expr.holders.VarCharHolder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)