[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803088#comment-14803088
 ] 

ASF GitHub Bot commented on DRILL-3779:
---

Github user adeneche commented on the pull request:

https://github.com/apache/drill/pull/160#issuecomment-141121314
  
two small comments to address, apart from that LGTM, 
+1


> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3689) incorrect results : aggregate AVG returns wrong results over results returned by LEAD function.

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803200#comment-14803200
 ] 

Khurram Faraaz commented on DRILL-3689:
---

Are we going to report an overflow error to user or will we return some 
results. If we plan report an overflow error I can add a negative test, if we 
do not plan to report an overflow error then we may want to document this 
behavior.

> incorrect results : aggregate AVG returns wrong results over results returned 
> by LEAD function.
> ---
>
> Key: DRILL-3689
> URL: https://issues.apache.org/jira/browse/DRILL-3689
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
> Environment:  private-branch 
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> Aggregate AVG returns wrong results over results returned by LEAD function.
> results returned by Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) 
> OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) 
> sub_query;
> +-+
> | EXPR$0  |
> +-+
> | 2.35195986941647008E17  |
> +-+
> 1 row selected (0.264 seconds)
> {code}
> Explain plan for above query from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT  avg(lead_col1) FROM 
> (SELECT LEAD(col1) OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 
> FROM FEWRWSPQQ_101) sub_query;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02Project(EXPR$0=[CAST(/(CastHigh(CASE(=($1, 0), null, $0)), 
> $1)):ANY NOT NULL])
> 00-03  StreamAgg(group=[{}], agg#0=[$SUM0($0)], agg#1=[COUNT($0)])
> 00-04Project(w0$o0=[$3])
> 00-05  Window(window#0=[window(partition {2} order by [1] range 
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [LEAD($1)])])
> 00-06SelectionVectorRemover
> 00-07  Sort(sort0=[$2], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-08Project(T36¦¦*=[$0], col1=[$1], col7=[$2])
> 00-09  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/FEWRWSPQQ_101]], 
> selectionRoot=maprfs:/tmp/FEWRWSPQQ_101, numFiles=1, columns=[`*`]]])
> {code}
> results returned by Postgres
> {code}
> postgres=# SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) OVER(PARTITION BY 
> col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) sub_query;
>  avg 
> -
>  1157533190627124568
> (1 row)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3732) Drill leaks memory if external sort hits out of disk space exception

[Aman Sinha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Sinha updated DRILL-3732:
--
Assignee: Deneche A. Hakim  (was: Aman Sinha)

> Drill leaks memory if external sort hits out of disk space exception
> 
>
> Key: DRILL-3732
> URL: https://issues.apache.org/jira/browse/DRILL-3732
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: drillbit.log
>
>
> Ran into it when running CTAS with partition by.
> Here is what reproduction looks like:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk, 
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) 
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > case when columns[2] = '' then cast(null as 
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[3] = '' then cast(null as 
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[4] = '' then cast(null as 
> varchar(100)) else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[5] = '' then cast(null as 
> varchar(100)) else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as 
> varchar(100)) else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[8] = '' then cast(null as 
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . >  `store_sales.dat` ss 
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  
> Expected private and shared pools to be set to initial values.  However, one 
> or more were not.  Stats are
>   zoneinitallocated   delta 
>   private 10009680512 319488 
>   shared  100010000.
> Fragment 1:21
> [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> Setup:
> single node
> 8GB direct memory
> 4GB heap memory
>store_sales.dat is a file from TPCDS SF100
> drillbit.log attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3732) Drill leaks memory if external sort hits out of disk space exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803191#comment-14803191
 ] 

ASF GitHub Bot commented on DRILL-3732:
---

Github user amansinha100 commented on the pull request:

https://github.com/apache/drill/pull/147#issuecomment-141143211
  
+1 .  Rest looks good.  Pls add the comment for the 'threw = false' 
assignment. 


> Drill leaks memory if external sort hits out of disk space exception
> 
>
> Key: DRILL-3732
> URL: https://issues.apache.org/jira/browse/DRILL-3732
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: drillbit.log
>
>
> Ran into it when running CTAS with partition by.
> Here is what reproduction looks like:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk, 
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) 
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > case when columns[2] = '' then cast(null as 
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[3] = '' then cast(null as 
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[4] = '' then cast(null as 
> varchar(100)) else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[5] = '' then cast(null as 
> varchar(100)) else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as 
> varchar(100)) else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . > case when columns[8] = '' then cast(null as 
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . >  `store_sales.dat` ss 
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  
> Expected private and shared pools to be set to initial values.  However, one 
> or more were not.  Stats are
>   zoneinitallocated   delta 
>   private 10009680512 319488 
>   shared  100010000.
> Fragment 1:21
> [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> Setup:
> single node
> 8GB direct memory
> 4GB heap memory
>store_sales.dat is a file from TPCDS SF100
> drillbit.log attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803061#comment-14803061
 ] 

ASF GitHub Bot commented on DRILL-3779:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/160#discussion_r39758626
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -478,28 +498,36 @@ public BatchGroup 
mergeAndSpill(LinkedList batchGroups) throws Schem
 break;
   }
   BatchGroup batch = batchGroups.pollLast();
+  assert batch != null;
--- End diff --

please add a message to the assertion, it will make our life easier if this 
assertion throws


> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3689) incorrect results : aggregate AVG returns wrong results over results returned by LEAD function.

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-3689:
--
Assignee: Deneche A. Hakim  (was: Khurram Faraaz)

> incorrect results : aggregate AVG returns wrong results over results returned 
> by LEAD function.
> ---
>
> Key: DRILL-3689
> URL: https://issues.apache.org/jira/browse/DRILL-3689
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
> Environment:  private-branch 
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> Aggregate AVG returns wrong results over results returned by LEAD function.
> results returned by Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) 
> OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) 
> sub_query;
> +-+
> | EXPR$0  |
> +-+
> | 2.35195986941647008E17  |
> +-+
> 1 row selected (0.264 seconds)
> {code}
> Explain plan for above query from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT  avg(lead_col1) FROM 
> (SELECT LEAD(col1) OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 
> FROM FEWRWSPQQ_101) sub_query;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02Project(EXPR$0=[CAST(/(CastHigh(CASE(=($1, 0), null, $0)), 
> $1)):ANY NOT NULL])
> 00-03  StreamAgg(group=[{}], agg#0=[$SUM0($0)], agg#1=[COUNT($0)])
> 00-04Project(w0$o0=[$3])
> 00-05  Window(window#0=[window(partition {2} order by [1] range 
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [LEAD($1)])])
> 00-06SelectionVectorRemover
> 00-07  Sort(sort0=[$2], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-08Project(T36¦¦*=[$0], col1=[$1], col7=[$2])
> 00-09  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/FEWRWSPQQ_101]], 
> selectionRoot=maprfs:/tmp/FEWRWSPQQ_101, numFiles=1, columns=[`*`]]])
> {code}
> results returned by Postgres
> {code}
> postgres=# SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) OVER(PARTITION BY 
> col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) sub_query;
>  avg 
> -
>  1157533190627124568
> (1 row)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803055#comment-14803055
 ] 

ASF GitHub Bot commented on DRILL-3779:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/160#discussion_r39757806
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
 ---
@@ -114,6 +119,12 @@
   private final String fileName;
   private int firstSpillBatchCount = 0;
 
+  /**
+   * The copier uses the COPER_BATCH_MEM_LIMIT to estimate the target
--- End diff --

small typo `COPIER_...` instead of `COPER_...`


> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[Aman Sinha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Sinha updated DRILL-3779:
--
Assignee: Deneche A. Hakim  (was: Aman Sinha)

> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Deneche A. Hakim
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803049#comment-14803049
 ] 

ASF GitHub Bot commented on DRILL-3779:
---

GitHub user amansinha100 opened a pull request:

https://github.com/apache/drill/pull/160

DRILL-3779: Fix NPE in mergeAndSpill().  Add more debug logging messa…

…ges.  Make copier per-batch memory limit power of 2.

Get rid of some warnings.  Add a few comments.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amansinha100/incubator-drill external_sort2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/160.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #160


commit 5eac063b5a0c6a149d41a26889d6636dd9bebf1b
Author: Aman Sinha 
Date:   2015-09-17T08:29:38Z

DRILL-3779: Fix NPE in mergeAndSpill().  Add more debug logging messages.  
Make copier per-batch memory limit power of 2.
Get rid of some warnings.  Add a few comments.




> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803366#comment-14803366
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/159#issuecomment-141176948
  
I worked around the fallback structure of option managers, which is wrong. 
I will update the pull request soon.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3689) incorrect results : aggregate AVG returns wrong results over results returned by LEAD function.

[Deneche A. Hakim (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803259#comment-14803259
 ] 

Deneche A. Hakim commented on DRILL-3689:
-

Checking for overflow can have serious performance impact. In the short term, 
we are not planing to detect overflows

> incorrect results : aggregate AVG returns wrong results over results returned 
> by LEAD function.
> ---
>
> Key: DRILL-3689
> URL: https://issues.apache.org/jira/browse/DRILL-3689
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
> Environment:  private-branch 
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> Aggregate AVG returns wrong results over results returned by LEAD function.
> results returned by Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) 
> OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) 
> sub_query;
> +-+
> | EXPR$0  |
> +-+
> | 2.35195986941647008E17  |
> +-+
> 1 row selected (0.264 seconds)
> {code}
> Explain plan for above query from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT  avg(lead_col1) FROM 
> (SELECT LEAD(col1) OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 
> FROM FEWRWSPQQ_101) sub_query;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02Project(EXPR$0=[CAST(/(CastHigh(CASE(=($1, 0), null, $0)), 
> $1)):ANY NOT NULL])
> 00-03  StreamAgg(group=[{}], agg#0=[$SUM0($0)], agg#1=[COUNT($0)])
> 00-04Project(w0$o0=[$3])
> 00-05  Window(window#0=[window(partition {2} order by [1] range 
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [LEAD($1)])])
> 00-06SelectionVectorRemover
> 00-07  Sort(sort0=[$2], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-08Project(T36¦¦*=[$0], col1=[$1], col7=[$2])
> 00-09  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/FEWRWSPQQ_101]], 
> selectionRoot=maprfs:/tmp/FEWRWSPQQ_101, numFiles=1, columns=[`*`]]])
> {code}
> results returned by Postgres
> {code}
> postgres=# SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) OVER(PARTITION BY 
> col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) sub_query;
>  avg 
> -
>  1157533190627124568
> (1 row)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14803374#comment-14803374
 ] 

Victoria Markman commented on DRILL-3238:
-

I only marked this case as critical, because it is wrong result. However, all 
the straightforward cases work, I don't think it is that imprortant.

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3796) Provide an iterator over record batches

[amit hadke (JIRA)]

amit hadke created DRILL-3796:
-

 Summary: Provide an iterator over record batches
 Key: DRILL-3796
 URL: https://issues.apache.org/jira/browse/DRILL-3796
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: amit hadke
Assignee: amit hadke


In order to simplify merge join operator provide an iterator over record batch 
that has seek(), mark(), reset().




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman updated DRILL-3238:

Priority: Critical  (was: Major)

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3689) incorrect results : aggregate AVG returns wrong results over results returned by LEAD function.

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim resolved DRILL-3689.
-
Resolution: Not A Problem

> incorrect results : aggregate AVG returns wrong results over results returned 
> by LEAD function.
> ---
>
> Key: DRILL-3689
> URL: https://issues.apache.org/jira/browse/DRILL-3689
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0
> Environment:  private-branch 
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> Aggregate AVG returns wrong results over results returned by LEAD function.
> results returned by Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) 
> OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) 
> sub_query;
> +-+
> | EXPR$0  |
> +-+
> | 2.35195986941647008E17  |
> +-+
> 1 row selected (0.264 seconds)
> {code}
> Explain plan for above query from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT  avg(lead_col1) FROM 
> (SELECT LEAD(col1) OVER(PARTITION BY col7 ORDER BY col1) lead_col1 , col7 
> FROM FEWRWSPQQ_101) sub_query;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(EXPR$0=[$0])
> 00-02Project(EXPR$0=[CAST(/(CastHigh(CASE(=($1, 0), null, $0)), 
> $1)):ANY NOT NULL])
> 00-03  StreamAgg(group=[{}], agg#0=[$SUM0($0)], agg#1=[COUNT($0)])
> 00-04Project(w0$o0=[$3])
> 00-05  Window(window#0=[window(partition {2} order by [1] range 
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [LEAD($1)])])
> 00-06SelectionVectorRemover
> 00-07  Sort(sort0=[$2], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-08Project(T36¦¦*=[$0], col1=[$1], col7=[$2])
> 00-09  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/FEWRWSPQQ_101]], 
> selectionRoot=maprfs:/tmp/FEWRWSPQQ_101, numFiles=1, columns=[`*`]]])
> {code}
> results returned by Postgres
> {code}
> postgres=# SELECT  avg(lead_col1) FROM (SELECT LEAD(col1) OVER(PARTITION BY 
> col7 ORDER BY col1) lead_col1 , col7 FROM FEWRWSPQQ_101) sub_query;
>  avg 
> -
>  1157533190627124568
> (1 row)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman reopened DRILL-3238:
-

Sean,

I just realized that test failed for me and query now returns wrong result.
Please see below:

-- wrong result caused by a wrong plan (notice two window operators)
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over  (w),
. . . . . . . . . . . . > count(*) over w
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2
. . . . . . . . . . . . > window w as (partition by a2 order by a2);
+-+-+-+
| a2  | EXPR$1  | EXPR$2  |
+-+-+-+
| 0   | 0   | 1   |
| 1   | 1   | 1   |
| 2   | 7   | 3   |
| 2   | 7   | 3   |
| 2   | 7   | 3   |
| 3   | 10  | 1   |
| 4   | 14  | 1   |
| 5   | 19  | 1   |
| 6   | 25  | 1   |
| 7   | 39  | 2   |
| 7   | 39  | 2   |
| 8   | 47  | 1   |
| 9   | 56  | 1   |
+-+-+-+
13 rows selected (0.305 seconds)

0: jdbc:drill:schema=dfs> explain plan for select a2, sum(a2) over(w), count(*) 
over w from t2 window w as (partition by a2 order by a2);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(a2=[$0], EXPR$1=[$1], EXPR$2=[$2])
00-02Window(window#0=[window(partition {0} order by [0] range between 
UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT()])])
00-03  SelectionVectorRemover
00-04Sort(sort0=[$0], sort1=[$0], dir0=[ASC], dir1=[ASC])
00-05  Window(window#0=[window(partition {} order by [0] range 
between UNBOUNDED PRECEDING and CURRENT ROW aggs [SUM($0)])])
00-06SelectionVectorRemover
00-07  Sort(sort0=[$0], dir0=[ASC])
00-08Scan(groupscan=[ParquetGroupScan 
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/t2]], 
selectionRoot=maprfs:/drill/testdata/subqueries/t2, numFiles=1, 
columns=[`a2`]]])
{code}

-- correct
{code}
0: jdbc:drill:schema=dfs> select a2, sum(a2) over (partition by a2 order by a2) 
from t2;
+-+-+
| a2  | EXPR$1  |
+-+-+
| 0   | 0   |
| 1   | 1   |
| 2   | 6   |
| 2   | 6   |
| 2   | 6   |
| 3   | 3   |
| 4   | 4   |
| 5   | 5   |
| 6   | 6   |
| 7   | 14  |
| 7   | 14  |
| 8   | 8   |
| 9   | 9   |
+-+-+
13 rows selected (0.303 seconds)
{code}

--correct
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over (partition by a2 order by a2),
. . . . . . . . . . . . > count(*) over(partition by a2 order by a2)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2;
+-+-+-+
| a2  | EXPR$1  | EXPR$2  |
+-+-+-+
| 0   | 0   | 1   |
| 1   | 1   | 1   |
| 2   | 6   | 3   |
| 2   | 6   | 3   |
| 2   | 6   | 3   |
| 3   | 3   | 1   |
| 4   | 4   | 1   |
| 5   | 5   | 1   |
| 6   | 6   | 1   |
| 7   | 14  | 2   |
| 7   | 14  | 2   |
| 8   | 8   | 1   |
| 9   | 9   | 1   |
+-+-+-+
13 rows selected (0.324 seconds)
{code}

-- correct
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over w,
. . . . . . . . . . . . > count(*) over w
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2
. . . . . . . . . . . . > window w as (partition by a2 order by a2);
+-+-+-+
| a2  | EXPR$1  | EXPR$2  |
+-+-+-+
| 0   | 0   | 1   |
| 1   | 1   | 1   |
| 2   | 6   | 3   |
| 2   | 6   | 3   |
| 2   | 6   | 3   |
| 3   | 3   | 1   |
| 4   | 4   | 1   |
| 5   | 5   | 1   |
| 6   | 6   | 1   |
| 7   | 14  | 2   |
| 7   | 14  | 2   |
| 8   | 8   | 1   |
| 9   | 9   | 1   |
+-+-+-+
13 rows selected (0.29 seconds)
{code}

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804471#comment-14804471
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39796601
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/InMemoryOptionManager.java
 ---
@@ -35,6 +37,20 @@
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+throw UserException.unsupportedError()
+  .message("This manager does not support deleting an option.")
+  .build(logger);
+  }
+
+  @Override
+  public void deleteAllOptions(final OptionValue.OptionType type) {
+throw UserException.unsupportedError()
--- End diff --

same here


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804469#comment-14804469
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39796579
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/InMemoryOptionManager.java
 ---
@@ -35,6 +37,20 @@
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+throw UserException.unsupportedError()
--- End diff --

Why throw a UserException here ? is the error message relevant to the user 
? do we even expect this method to be called ? 


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-3779:

Assignee: Aman Sinha  (was: Deneche A. Hakim)

> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804481#comment-14804481
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39797379
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
+SystemOptionManager.getValidator(name);
+  } catch (final IllegalArgumentException e) {
+throw UserException.validationError()
--- End diff --

you can just use:
```
throw UserException.validationError(e).build(logger);
```


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804513#comment-14804513
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39799140
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 ---
@@ -241,6 +243,31 @@ public void setOption(final OptionValue value) {
   }
 
   @Override
+  public void deleteOption(final String name, OptionType type) {
+assert type == OptionType.SYSTEM;
+try { // ensure option exists
+  getValidator(name);
+} catch (final IllegalArgumentException e) {
+  throw UserException.validationError()
--- End diff --

you can just use:
```
throw UserException.validationError(e).build(logger);
```


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804601#comment-14804601
 ] 

Chris Westin edited comment on DRILL-2166 at 9/17/15 9:58 PM:
--

Whatever it was seems to be fixed now. I get this result:

bq.
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>



was (Author: cwestin):
Whatever it was seems to be fixed now. I get this result:

{{
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
}}

> left join with complex type throw 

[jira] [Comment Edited] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804601#comment-14804601
 ] 

Chris Westin edited comment on DRILL-2166 at 9/17/15 9:57 PM:
--

Whatever it was seems to be fixed now. I get this result:

{{
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
}}


was (Author: cwestin):
Whatever it was seems to be fixed now. I get this result:

{{monospaced}}
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
{{monospaced}}

> left join with complex 

[jira] [Comment Edited] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804601#comment-14804601
 ] 

Chris Westin edited comment on DRILL-2166 at 9/17/15 9:58 PM:
--

Whatever it was seems to be fixed now. I get this result:

0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>



was (Author: cwestin):
Whatever it was seems to be fixed now. I get this result:

{quote}
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
{quote}

> left join with complex type throw 

[jira] [Comment Edited] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804601#comment-14804601
 ] 

Chris Westin edited comment on DRILL-2166 at 9/17/15 9:58 PM:
--

Whatever it was seems to be fixed now. I get this result:

{quote}
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
{quote}


was (Author: cwestin):
Whatever it was seems to be fixed now. I get this result:

bq.
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>


> left join with complex type throw 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804626#comment-14804626
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on the pull request:

https://github.com/apache/drill/pull/159#issuecomment-141251928
  
I have added a note to the SessionOptionManager that *the effects of 
deleting a short lived option are undefined* because that might require 
slightly extensive changes, which is out of scope for this JIRA.

@jacques-n please review the changes in *CompoundIdentifierConverter* class.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1976) Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804632#comment-14804632
 ] 

Chris Westin commented on DRILL-1976:
-

[~rkins], how is the generation script supposed to be used? I tried

./wide-strings.sh > widestrings.json

But the shell is seeing a lot of
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence
tr: Illegal byte sequence

And what json records do appear in the file do not have wide character string 
columns:
{
"id":14,
"str_fixed":"wx",
"str_var":"",
"str_empty":"",
"str_null":"",
"str_empty_null":"",
"str_var_null_empty":"",
"str_fixed_null_empty":"U",
"tinyint_var":93,
"dec_var_prec5_sc2":.
}
{
"id":15,
"str_fixed":"",
"str_var":"",
"str_empty":"",
"str_null":null,
"str_empty_null":null,
"str_var_null_empty":"",
"str_fixed_null_empty":"U",
"tinyint_var":-105,
"dec_var_prec5_sc2":-.0
}



> Possible Memory Leak in drill jdbc client when dealing with wide columns 
> (5000 chars long)
> --
>
> Key: DRILL-1976
> URL: https://issues.apache.org/jira/browse/DRILL-1976
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Rahul Challapalli
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: wide-strings.sh
>
>
> git.commit.id.abbrev=b491cdb
> I am seeing an execution failure when I execute the same query multiple times 
> (<10). The data file contains 9 columns out of which 7 are wide strings 
> (4000-5000 chars long)
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> {code}
> Below are my memory settings :
> {code}
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> {code}
> Error From the JDBC client
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> Exception in pipeline.  Closing channel between local /10.10.100.190:38179 
> and remote qa-node191.qa.lab/10.10.100.191:31010
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct 
> buffer memory
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658)
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>   at 
> io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
>   at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
>   at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
>   at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
>   at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
>   at 
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
>   at 

[jira] [Updated] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-2748:
--
Assignee: Aman Sinha  (was: Jinfeng Ni)

> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-2748:
--
Attachment: (was: 
0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch)

> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804528#comment-14804528
 ] 

Jinfeng Ni commented on DRILL-2748:
---

[~amansinha100], could you please review the patch? Thanks!



> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-2418) Memory leak during execution if comparison function is not found

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin closed DRILL-2418.
---
Resolution: Fixed

> Memory leak during execution if comparison function is not found
> 
>
> Key: DRILL-2418
> URL: https://issues.apache.org/jira/browse/DRILL-2418
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: cast_tbl_1.parquet, cast_tbl_2.parquet, 
> not_supported_cast.txt
>
>
> While testing implicit cast during join, I ran into an issue where if you run 
> a query that throws an exception during execution, eventually, if you run 
> enough of those, drill will run out of memory.
> Here is a query example:
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.c_float = b.c_time
>  failed: RemoteRpcException: Failure while running fragment., Failure finding 
> function that runtime code generation expected.  Signature: 
> compare_to_nulls_high( TIME:OPTIONAL, FLOAT4:OPTIONAL ) returns INT:REQUIRED 
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> Test_Failed: 2015/03/10 18:34:15.0015 - Failed to execute.
> {code}
> If you set planner.slice_target to 1, you hit out of memory after about ~40 
> or so of such failures on my cluster.
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.d38 = b.c_double
> Query failed: OutOfMemoryException: You attempted to create a new child 
> allocator with initial reservation 300 but only 916199 bytes of memory 
> were available.
> {code}
> From the drillbit.log
> {code}
> 2015-03-10 18:34:34,588 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out 
> of 1 using 1 threads. Time: 1ms total, 1.190007ms avg, 1ms max.
> 2015-03-10 18:34:34,591 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out 
> of 1 using 1 threads. Time: 0ms total, 0.953679ms avg, 0ms max.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> atsqa4-136.qa.lab.  Skipping affinity to that host.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: 
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.609586ms avg, 1ms max.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> atsqa4-136.qa.lab.  Skipping affinity to that host.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: 
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.270340ms avg, 1ms max.
> 2015-03-10 18:34:34,684 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - State change requested.  PENDING --> 
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception 
> during fragment initialization: Failure while getting memory allocator for 
> fragment.
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:195) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_71]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
> Failure while getting memory allocator for fragment.
> at 
> org.apache.drill.exec.ops.FragmentContext.(FragmentContext.java:119) 
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.setupRootFragment(Foreman.java:535)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:307) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:511) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 

[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804541#comment-14804541
 ] 

Jinfeng Ni commented on DRILL-2748:
---

The reason that the unit test case I added in the first path worked : the 
filter is on partition column. The filter pushdown lead to partition pruning, 
which would lead to reduction in the scan cost. Therefore, the new plan with 
filter push down is estimated to have lower cost.


> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[Sudheesh Katkam (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804598#comment-14804598
 ] 

Sudheesh Katkam commented on DRILL-1065:


Also, adding:
SET `option name`
RESET `option name`
RESET ALL

as discussed 
[here|http://mail-archives.apache.org/mod_mbox/calcite-dev/201507.mbox/%3ccampyv7c9ynatuk-2ufh_suddmc6d5-e4jd1_taqmnuoifz1...@mail.gmail.com%3E]
 with an assumption that the scope is SESSION.

> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-1976) Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-1976:
-
Attachment: wide-strings-mod.sh

[~cwestin] I guess you are running it on a mac. From my linux box the script 
ran fine. 

I made a little change and uploaded a new file. Can you try it out?

> Possible Memory Leak in drill jdbc client when dealing with wide columns 
> (5000 chars long)
> --
>
> Key: DRILL-1976
> URL: https://issues.apache.org/jira/browse/DRILL-1976
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Rahul Challapalli
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: wide-strings-mod.sh, wide-strings.sh
>
>
> git.commit.id.abbrev=b491cdb
> I am seeing an execution failure when I execute the same query multiple times 
> (<10). The data file contains 9 columns out of which 7 are wide strings 
> (4000-5000 chars long)
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> {code}
> Below are my memory settings :
> {code}
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> {code}
> Error From the JDBC client
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> Exception in pipeline.  Closing channel between local /10.10.100.190:38179 
> and remote qa-node191.qa.lab/10.10.100.191:31010
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct 
> buffer memory
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658)
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>   at 
> io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
>   at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
>   at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
>   at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
>   at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
>   at 
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
>   at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:600)
>   at 
> io.netty.buffer.UnsafeDirectLittleEndian.writeBytes(UnsafeDirectLittleEndian.java:25)
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:144)
>   ... 13 more
> Channel closed between local /10.10.100.190:38179 and remote 
> qa-node191.qa.lab/10.10.100.191:31010
> Channel is closed, discarding remaining 255231 byte(s) in buffer.
> {code}
> The logs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804505#comment-14804505
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39798736
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
+SystemOptionManager.getValidator(name);
+  } catch (final IllegalArgumentException e) {
+throw UserException.validationError()
+  .message(e.getMessage())
+  .build(logger);
+  }
+  options.remove(name);
+  shortLivedOptions.remove(name);
+} else {
+  fallback.deleteOption(name, type);
--- End diff --

so if I change an option both at the _session_ and _system_ levels, then I 
_reset_ the same option at the _system_ level it will leave it changed at the 
_session_ level. Is this expected ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804504#comment-14804504
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39798636
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
--- End diff --

You only ensure the option exists when `type == SESSION`. How about 
_SYSTEM_, is it done by the underlying _fallback_ OptionManager ?


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-2748:
--
Attachment: 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch

> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch, 
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804587#comment-14804587
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39803498
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
+SystemOptionManager.getValidator(name);
+  } catch (final IllegalArgumentException e) {
+throw UserException.validationError()
+  .message(e.getMessage())
+  .build(logger);
+  }
+  options.remove(name);
+  shortLivedOptions.remove(name);
+} else {
+  fallback.deleteOption(name, type);
--- End diff --

I'd say yes. Session options always override system options (default or 
not).


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804583#comment-14804583
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39803386
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
--- End diff --

Yes, the fallback for a SessionOptionManager is a SystemOptionManager.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3797) Filter not pushed down as part of scan (for JSON data)

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-3797:
-

 Summary: Filter not pushed down as part of scan (for JSON data)
 Key: DRILL-3797
 URL: https://issues.apache.org/jira/browse/DRILL-3797
 Project: Apache Drill
  Issue Type: New Feature
  Components: Execution - Relational Operators
Affects Versions: 1.2.0
Reporter: Khurram Faraaz
Assignee: Chris Westin
Priority: Minor


Filter is not part of the scan operator.

{code}
0: jdbc:drill:schema=dfs.tmp> explain plan for select key1, key2 from 
`twoKeyJsn.json` where key1 = 1.26807240856E8;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01  Project(key1=[$0], key2=[$1])
00-02SelectionVectorRemover
00-03  Filter(condition=[=($0, 1.26807240856E8)])
00-04Scan(groupscan=[EasyGroupScan 
[selectionRoot=maprfs:/tmp/twoKeyJsn.json, numFiles=1, columns=[`key1`, 
`key2`], files=[maprfs:///tmp/twoKeyJsn.json]]])
{code}

{code}

Here is snippet of data from JSON file
root@centos-01 ~]# head -n 10 twoKeyJsn.json
{"key1":1296815267.3,"key2":"d"}
{"key1":46736552.9012,"key2":"c"}
{"key1":93968206.5896,"key2":"b"}
{"key1":1015801729.33,"key2":"d"}
{"key1":49878.641,"key2":"1"}
{"key1":152391833.107,"key2":"1"}
{"key1":731290386.917,"key2":"a"}
{"key1":692726688.161,"key2":"d"}
{"key1":1123835226.54,"key2":"a"}
{"key1":126807240.856,"key2":"1"}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin resolved DRILL-2166.
-
Resolution: Fixed

> left join with complex type throw ClassTransformationException
> --
>
> Key: DRILL-2166
> URL: https://issues.apache.org/jira/browse/DRILL-2166
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 0.8.0
>Reporter: Chun Chang
>Assignee: Chris Westin
> Fix For: 1.2.0
>
>
> #Thu Jan 29 18:00:57 EST 2015
> git.commit.id.abbrev=09f7fb2
> Dataset can be downloaded from 
> https://s3.amazonaws.com/apache-drill/files/complex.json.gz
> The following query caused the exception:
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select a.id, a.soa, b.sfa[0], 
> b.soa[1] from `complex.json` a left outer join `complex.json` b on 
> a.sia[0]=b.sia[0] order by a.id limit 20;
> Query failed: RemoteRpcException: Failure while running fragment., Line 35, 
> Column 32: No applicable constructor/method found for actual parameters "int, 
> int, org.apache.drill.exec.vector.complex.MapVector"; candidates are: "public 
> void org.apache.drill.exec.vector.NullableTinyIntVector.copyFromSafe(int, 
> int, org.apache.drill.exec.vector.NullableTinyIntVector)", "public void 
> org.apache.drill.exec.vector.NullableTinyIntVector.copyFromSafe(int, int, 
> org.apache.drill.exec.vector.TinyIntVector)" [ 
> fbf47be8-b5fe-4d56-9488-15d45d4224e4 on qa-node117.qa.lab:31010 ]
> [ fbf47be8-b5fe-4d56-9488-15d45d4224e4 on qa-node117.qa.lab:31010 ]
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}
> stack from drill bit.log
> {code}
> 2015-02-04 13:37:22,117 [2b2d6eee-105b-5544-9111-83a3a356285d:frag:2:6] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing 
> fragment
> org.apache.drill.common.exceptions.DrillRuntimeException: 
> org.apache.drill.exec.exception.SchemaChangeException: 
> org.apache.drill.exec.exception.ClassTransformationException: 
> java.util.concurrent.ExecutionException: 
> org.apache.drill.exec.exception.ClassTransformationException: Failure 
> generating transformation classes for value:
> package org.apache.drill.exec.test.generated;
> import org.apache.drill.exec.exception.SchemaChangeException;
> import org.apache.drill.exec.ops.FragmentContext;
> import org.apache.drill.exec.record.RecordBatch;
> import org.apache.drill.exec.record.VectorContainer;
> import org.apache.drill.exec.vector.NullableBigIntVector;
> import org.apache.drill.exec.vector.NullableFloat8Vector;
> import org.apache.drill.exec.vector.NullableTinyIntVector;
> import org.apache.drill.exec.vector.complex.MapVector;
> import org.apache.drill.exec.vector.complex.RepeatedMapVector;
> {code}
> from forman drill bit.log
> {code}
> 2015-02-04 13:37:22,189 [BitServer-5] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in pipeline.  Closing channel 
> between local /10.10.100.117:31012 and remote /10.10.100.120:56250
> io.netty.handler.codec.DecoderException: java.lang.NullPointerException
>   at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
>  [netty-codec-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>  [netty-codec-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
>  [netty-codec-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>  [netty-transport-4.0.24.Final.jar:4.0.24.Final]
>   at 
> 

[jira] [Commented] (DRILL-3229) Create a new EmbeddedVector

[Steven Phillips (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804637#comment-14804637
 ] 

Steven Phillips commented on DRILL-3229:


Basic design outline:

A Union type represents a field where the type can vary between records. The 
data for a field of type Union will be stored in a UnionVector.

h4. UnionVector
Internally uses a MapVector to hold the vectors for the various types. 
The types include all of the MinorTypes, including List and Map.
For example, the internal MapVector will have a subfield named 
"bigInt", which will refer to a NullableBigIntVector.

In addition to the vectors corresponding to the minor types, there will 
be two additional fields, both represented by UInt1Vectors. These are
"bits" and "types", which will represent the nullability and types of 
the underlying data. The "bits" vector will work the same way it works in other
nullable vectors. The "types" vector will store the number 
corresponding to the value of the MinorType as defined in the protobuf 
definition. There
will be mutator methods for setting null and type.

h4. UnionWriter
The UnionWriter implements and overwrites all of the methods of 
FieldWriter. It holds field writers corresponding to each of the types included 
in the underly
UnionVector, and delegates the method calls for each type to the 
corresponding writer. For example, the BigIntWriter interface:

{code}
public interface BigIntWriter extends BaseWriter {
  public void write(BigIntHolder h);

  public void writeBigInt(long value);
}
{code}
UnionWriter overwrites these methods:

{code}
@Override
  public void writeBigInt(long value) {
data.getMutator().setType(idx(), MinorType.BIGINT);
data.getMutator().setNotNull(idx());
getBigIntWriter().setPosition(idx());
getBigIntWriter().writeBigInt(value);
  }

@Override
  public void writeBigInt(BigIntHolder h) {
data.getMutator().setType(idx(), MinorType.BIGINT);
data.getMutator().setNotNull(idx());
getBigIntWriter().setPosition(idx());
getBigIntWriter().writeBigInt(holder.value);
  }
{code}

This requires users of the interface to go through the UnionWriter, 
rather than using the underlying BigIntWriter directly. Otherwise, the "type" 
and "bits" vector would not get set correctly.

h4. UnionReader
Much the same as the UnionWriter, the UnionReader overwrites the 
methods of FieldReader, and delegates to a corresponding specific FieldReader 
implementation depending on which type 
the current value is.

h4. UnionListVector
UnionListVector extends BaseRepeatedVector. It works much the same as 
other Repeated vectors; there is a data vector and an offset vector. The data 
vector in this case is a UnionVector.

h4. UnionListWriter
The UnionListWriter overrides all FieldWriter methods. When starting a 
new list, the startList() method is called. This calls the startNewValue(int 
index) method
of the underlying UnionListVector.Mutator. Subsequent calls to the 
ListWriter methods (such as bigint()), return the UnionListWriter itself, and 
calls to write are handled by calling
the appropriate method on the underlying UnionListVector.Mutator, which 
handles updating the offset vector.

In the case that the map() method is called (i.e. repeated map), the 
UnionListWriter is itself returned, but a state variable is updated to indicate 
that it should oeprate as a MapWriter.
While in MapWriter mode, calls to the MapWriter methods will also 
return the UnionListWriter itself, but will also update the field indicating 
what the name of the current field is.
Subsequent writes to the ScalarWriter methods will write to the 
underlying UnionVector using the UnionWriter interface.

For example,

{code}
UnionListWriter list;
...

list.startList();
list.map().bigInt("a").writeBigInt(1);
{code}

This code first indicates that a new list is starting. By doing this, 
the offset vector is correctly set. Calling map() sets the internal state of 
the writer to "MAP". bigInt("a") sets the current
field of the writer to "a", and writeBigInt(1) writes the value 1 to 
the underlying UnionVector.
Another example:

{code}
MapWriter mapWriter = list.map().map("a")
{code}

In this case, the final call to map("a") delegates to the underlying 
UnionWriter, and returns a new MapWriter, with the position set according to 
the current offset.

> Create a new EmbeddedVector
> ---
>
> Key: DRILL-3229
> URL: https://issues.apache.org/jira/browse/DRILL-3229
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Codegen, Execution - Data Types, Execution - 
> Relational Operators, Functions - Drill
>Reporter: 

[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Sean Hsuan-Yi Chu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804649#comment-14804649
 ] 

Sean Hsuan-Yi Chu commented on DRILL-3238:
--

Filed a calcite issue here: 
https://issues.apache.org/jira/browse/CALCITE-888
(Patch is available as pull request)

That parenthesis is used to do "window overlay" trick: For example, you can 
have a query like:
{code}
select sum(c) over(w order by c)
from t
window w as (partition by c);
{code}

In the select list, {color:red} over(w order by c) {color} would be expanded to 
{color:red} over(partition by c order by c) {color} 




> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804511#comment-14804511
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user adeneche commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39799014
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SessionOptionManager.java
 ---
@@ -46,6 +47,33 @@ public SessionOptionManager(final OptionManager 
systemOptions, final UserSession
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+if (type == OptionValue.OptionType.SESSION) {
+  try { // ensure option exists
+SystemOptionManager.getValidator(name);
+  } catch (final IllegalArgumentException e) {
+throw UserException.validationError()
+  .message(e.getMessage())
+  .build(logger);
+  }
+  options.remove(name);
+  shortLivedOptions.remove(name);
+} else {
+  fallback.deleteOption(name, type);
+}
+  }
+
+  @Override
+  public void deleteAllOptions(final OptionValue.OptionType type) {
+if (type == OptionValue.OptionType.SESSION) {
+  options.clear();
+  shortLivedOptions.clear();
+} else {
+  fallback.deleteAllOptions(type);
--- End diff --

same here


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804523#comment-14804523
 ] 

Jinfeng Ni commented on DRILL-2748:
---

DrillAggregateRel, which is the logical aggregate operator, currently call 
makeTinyCost(), regardless how many rows the input has, how many aggregation 
functions it contains.  That lead to the costing estimation could not reflect 
the saving, if the filter is push past the aggregate operator. 

The fix is to improve the cost estimation for logical aggregate. Similar to 
join operator, we use the same formula for HashAgg and logical aggregate, under 
the assumption that HashAgg is more likely to happen in the physical planning.



> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: 
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into 
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1, 
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null, 
> $2)), $3)):ANY NOT NULL])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)], 
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, 
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select 
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01  Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05  HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]], 
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`, 
> `c1`]]])
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2418) Memory leak during execution if comparison function is not found

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804537#comment-14804537
 ] 

Chris Westin commented on DRILL-2418:
-

I simulated the old error by having it signal the old error (in 
FunctionGenerationHelper.getFunctionExpression()), with a query that gets past 
the new type cast detection (I hard wired the error), and used the query that 
gets further that you gave me:

select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.c_varchar = b.c_varchar;

It signaled the old error, and there was no memory leak. So it looks like the 
requisite cleanup handling for an error there has been added.

> Memory leak during execution if comparison function is not found
> 
>
> Key: DRILL-2418
> URL: https://issues.apache.org/jira/browse/DRILL-2418
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: cast_tbl_1.parquet, cast_tbl_2.parquet, 
> not_supported_cast.txt
>
>
> While testing implicit cast during join, I ran into an issue where if you run 
> a query that throws an exception during execution, eventually, if you run 
> enough of those, drill will run out of memory.
> Here is a query example:
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.c_float = b.c_time
>  failed: RemoteRpcException: Failure while running fragment., Failure finding 
> function that runtime code generation expected.  Signature: 
> compare_to_nulls_high( TIME:OPTIONAL, FLOAT4:OPTIONAL ) returns INT:REQUIRED 
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> Test_Failed: 2015/03/10 18:34:15.0015 - Failed to execute.
> {code}
> If you set planner.slice_target to 1, you hit out of memory after about ~40 
> or so of such failures on my cluster.
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.d38 = b.c_double
> Query failed: OutOfMemoryException: You attempted to create a new child 
> allocator with initial reservation 300 but only 916199 bytes of memory 
> were available.
> {code}
> From the drillbit.log
> {code}
> 2015-03-10 18:34:34,588 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out 
> of 1 using 1 threads. Time: 1ms total, 1.190007ms avg, 1ms max.
> 2015-03-10 18:34:34,591 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out 
> of 1 using 1 threads. Time: 0ms total, 0.953679ms avg, 0ms max.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> atsqa4-136.qa.lab.  Skipping affinity to that host.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: 
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.609586ms avg, 1ms max.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> atsqa4-136.qa.lab.  Skipping affinity to that host.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps: 
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.270340ms avg, 1ms max.
> 2015-03-10 18:34:34,684 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - State change requested.  PENDING --> 
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception 
> during fragment initialization: Failure while getting memory allocator for 
> fragment.
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:195) 
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
>  [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_71]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: 
> Failure while getting memory allocator for fragment.
> at 
> org.apache.drill.exec.ops.FragmentContext.(FragmentContext.java:119) 
> 

[jira] [Commented] (DRILL-1065) Provide a reset command to reset an option to its default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804580#comment-14804580
 ] 

ASF GitHub Bot commented on DRILL-1065:
---

Github user sudheeshkatkam commented on a diff in the pull request:

https://github.com/apache/drill/pull/159#discussion_r39803317
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/InMemoryOptionManager.java
 ---
@@ -35,6 +37,20 @@
   }
 
   @Override
+  public void deleteOption(final String name, final OptionValue.OptionType 
type) {
+throw UserException.unsupportedError()
--- End diff --

Removing this in my new implementation.


> Provide a reset command to reset an option to its default value
> ---
>
> Key: DRILL-1065
> URL: https://issues.apache.org/jira/browse/DRILL-1065
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Reporter: Aman Sinha
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.2.0
>
>
> Within a session, currently we set configuration options and it would be very 
> useful to have a 'reset' command to reset the value of an option to its 
> default system value: 
>   ALTER SESSION RESET  
> If we don't want to add a new keyword for RESET, we could potentially 
> overload the SET command and allow the user to set to the 'default' value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-2166) left join with complex type throw ClassTransformationException

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804601#comment-14804601
 ] 

Chris Westin edited comment on DRILL-2166 at 9/17/15 9:59 PM:
--

Whatever it was seems to be fixed now. I get this result:

{noformat}
0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>
{noformat}


was (Author: cwestin):
Whatever it was seems to be fixed now. I get this result:

0: jdbc:drill:zk=local> select a.id, a.soa, b.sfa[0], b.soa[1] from 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/drill/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order 
by a.id limit 20;
select a.id, a.soa, b.sfa[0], b.soa[1] from dfs.`/Users/ 
cwestin/drill/DRILL-2166/complex.json` a left outer join 
dfs.`/Users/cwestin/dri 
ll/DRILL-2166/complex.json` b on a.sia[0]=b.sia[0] order by a.id limit 20;
++-+++
| id | soa | EXPR$2 | EXPR$3 |
++-+++
| 1 | [{"in":1},{"in":1,"fl":1.12345},{"in":1,"fl":10.12345,"nul":"not null"}, |
| 2 | [{"in":2},{"in":2,"fl":2.12345},{"in":2,"fl":20.12345,"nul":"not null"}, |
| 3 | [{"in":3},{"in":3,"fl":3.12345},{"in":3,"fl":30.12345,"nul":"not null"}, |
| 4 | [{"in":4},{"in":4,"fl":4.12345},{"in":4,"fl":40.12345,"nul":"not null"}, |
| 5 | [{"in":5},{"in":5,"fl":5.12345},{"in":5,"fl":50.12345,"nul":"not null"}, |
| 6 | [{"in":6},{"in":6,"fl":6.12345},{"in":6,"fl":60.12345,"nul":"not null"}, |
| 7 | [{"in":7},{"in":7,"fl":7.12345},{"in":7,"fl":70.12345},{"in":7,"fl":70.6 |
| 8 | [{"in":8},{"in":8,"fl":8.12345},{"in":8,"fl":80.12345},{"in":8,"fl":80.6 |
| 9 | [{"in":9},{"in":9,"fl":9.12345},{"in":9,"fl":90.12345,"nul":"not null"}, |
| 10 | [{"in":10},{"in":10,"fl":10.12345},{"in":10,"fl":100.12345},{"in":10,"f |
| 11 | [{"in":11},{"in":11,"fl":11.12345},{"in":11,"fl":110.12345},{"in":11,"f |
| 12 | [{"in":12},{"in":12,"fl":12.12345},{"in":12,"fl":120.12345,"nul":"not n |
| 13 | [{"in":13},{"in":13,"fl":13.12345},{"in":13,"fl":130.12345},{"in":13,"f |
| 14 | [{"in":14},{"in":14,"fl":14.12345},{"in":14,"fl":140.12345,"nul":"not n |
| 15 | [{"in":15},{"in":15,"fl":15.12345},{"in":15,"fl":150.12345,"nul":"not n |
| 16 | [{"in":16},{"in":16,"fl":16.12345},{"in":16,"fl":160.12345},{"in":16,"f |
| 17 | [{"in":17},{"in":17,"fl":17.12345},{"in":17,"fl":170.12345},{"in":17,"f |
| 18 | [{"in":18},{"in":18,"fl":18.12345},{"in":18,"fl":180.12345,"nul":"not n |
| 19 | [{"in":19},{"in":19,"fl":19.12345},{"in":19,"fl":190.12345,"nul":"not n |
| 20 | [{"in":20},{"in":20,"fl":20.12345},{"in":20,"fl":200.12345,"nul":"not n |
++-+++
20 rows selected (35.17 seconds)
0: jdbc:drill:zk=local>


> left join with complex type throw 

[jira] [Updated] (DRILL-3362) Implicit cast causes workspace variable to not be initialized

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin updated DRILL-3362:

Assignee: Steven Phillips  (was: Chris Westin)

> Implicit cast causes workspace variable to not be initialized
> -
>
> Key: DRILL-3362
> URL: https://issues.apache.org/jira/browse/DRILL-3362
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Reporter: Steven Phillips
>Assignee: Steven Phillips
> Fix For: 1.2.0
>
>
> In DRILL-3361, the "newPartitionValue" function is missing the implementation 
> for Bit type. 
> When implicit cast is not needed, the workspace variable is initialized in 
> the __DRILL_INIT__() function of the generated class. But when implicit cast 
> is added, this initialization does not happen. This results in a NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3362) Implicit cast causes workspace variable to not be initialized

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804621#comment-14804621
 ] 

Chris Westin commented on DRILL-3362:
-

[~sphillips], could we get a test case please?

> Implicit cast causes workspace variable to not be initialized
> -
>
> Key: DRILL-3362
> URL: https://issues.apache.org/jira/browse/DRILL-3362
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Reporter: Steven Phillips
>Assignee: Chris Westin
> Fix For: 1.2.0
>
>
> In DRILL-3361, the "newPartitionValue" function is missing the implementation 
> for Bit type. 
> When implicit cast is not needed, the workspace variable is initialized in 
> the __DRILL_INIT__() function of the generated class. But when implicit cast 
> is added, this initialization does not happen. This results in a NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-3228) Implement Embedded Type

[Steven Phillips (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Phillips reassigned DRILL-3228:
--

Assignee: Steven Phillips  (was: Jacques Nadeau)

> Implement Embedded Type
> ---
>
> Key: DRILL-3228
> URL: https://issues.apache.org/jira/browse/DRILL-3228
> Project: Apache Drill
>  Issue Type: Task
>  Components: Execution - Codegen, Execution - Data Types, Execution - 
> Relational Operators, Functions - Drill
>Reporter: Jacques Nadeau
>Assignee: Steven Phillips
> Fix For: 1.3.0
>
>
> An Umbrella for the implementation of Embedded types within Drill.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-3229) Create a new EmbeddedVector

[Steven Phillips (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Phillips reassigned DRILL-3229:
--

Assignee: Steven Phillips  (was: Jacques Nadeau)

> Create a new EmbeddedVector
> ---
>
> Key: DRILL-3229
> URL: https://issues.apache.org/jira/browse/DRILL-3229
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Execution - Codegen, Execution - Data Types, Execution - 
> Relational Operators, Functions - Drill
>Reporter: Jacques Nadeau
>Assignee: Steven Phillips
> Fix For: Future
>
>
> Embedded Vector will leverage a binary encoding for holding information about 
> type for each individual field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1976) Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)

[Chris Westin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804845#comment-14804845
 ] 

Chris Westin commented on DRILL-1976:
-

The new script didn't work on OSX either. I ftp'ed the original to a linux VM, 
generated the data file, and then ftp'ed the data file back.

Once on OSX, I set up the memory parameters as you specified above. I was able 
to run the query 11 times in a row without errors, and shut down cleanly 
without any memory leaks reported. (This is on current master, which now has 
the majority of the fixes spawned by the new allocator, but not the new 
allocator itself yet). So it looks like whatever it was is fixed.

> Possible Memory Leak in drill jdbc client when dealing with wide columns 
> (5000 chars long)
> --
>
> Key: DRILL-1976
> URL: https://issues.apache.org/jira/browse/DRILL-1976
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Rahul Challapalli
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: wide-strings-mod.sh, wide-strings.sh
>
>
> git.commit.id.abbrev=b491cdb
> I am seeing an execution failure when I execute the same query multiple times 
> (<10). The data file contains 9 columns out of which 7 are wide strings 
> (4000-5000 chars long)
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> {code}
> Below are my memory settings :
> {code}
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> {code}
> Error From the JDBC client
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> Exception in pipeline.  Closing channel between local /10.10.100.190:38179 
> and remote qa-node191.qa.lab/10.10.100.191:31010
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct 
> buffer memory
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658)
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>   at 
> io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
>   at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
>   at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
>   at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
>   at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
>   at 
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
>   at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:600)
>   at 
> io.netty.buffer.UnsafeDirectLittleEndian.writeBytes(UnsafeDirectLittleEndian.java:25)
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:144)
>   ... 13 more
> Channel closed between local /10.10.100.190:38179 and remote 
> 

[jira] [Resolved] (DRILL-1976) Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)

[Chris Westin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Westin resolved DRILL-1976.
-
Resolution: Fixed

> Possible Memory Leak in drill jdbc client when dealing with wide columns 
> (5000 chars long)
> --
>
> Key: DRILL-1976
> URL: https://issues.apache.org/jira/browse/DRILL-1976
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Rahul Challapalli
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: wide-strings-mod.sh, wide-strings.sh
>
>
> git.commit.id.abbrev=b491cdb
> I am seeing an execution failure when I execute the same query multiple times 
> (<10). The data file contains 9 columns out of which 7 are wide strings 
> (4000-5000 chars long)
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> {code}
> Below are my memory settings :
> {code}
> DRILL_MAX_DIRECT_MEMORY="32G"
> DRILL_MAX_HEAP="4G"
> {code}
> Error From the JDBC client
> {code}
> select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select 
> str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on 
> ws.tinyint_var = sub.max_ti
> Exception in pipeline.  Closing channel between local /10.10.100.190:38179 
> and remote qa-node191.qa.lab/10.10.100.191:31010
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct 
> buffer memory
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
>   at java.nio.Bits.reserveMemory(Bits.java:658)
>   at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
>   at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>   at 
> io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
>   at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
>   at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
>   at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
>   at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
>   at 
> io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
>   at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:600)
>   at 
> io.netty.buffer.UnsafeDirectLittleEndian.writeBytes(UnsafeDirectLittleEndian.java:25)
>   at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:144)
>   ... 13 more
> Channel closed between local /10.10.100.190:38179 and remote 
> qa-node191.qa.lab/10.10.100.191:31010
> Channel is closed, discarding remaining 255231 byte(s) in buffer.
> {code}
> The logs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-2737) Sqlline throws Runtime exception when JDBC ResultSet throws a SQLException

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz closed DRILL-2737.
-

> Sqlline throws Runtime exception when JDBC ResultSet throws a SQLException
> --
>
> Key: DRILL-2737
> URL: https://issues.apache.org/jira/browse/DRILL-2737
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - CLI
>Reporter: Parth Chandra
>Assignee: Parth Chandra
> Fix For: 1.2.0
>
> Attachments: DRILL-2737.patch
>
>
> This is a tracking bug to provide a patch to Sqlline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2451) JDBC : Connection.commit throws an UnsupportedOperationException

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804848#comment-14804848
 ] 

Khurram Faraaz commented on DRILL-2451:
---

I used the below snippet to verify, however I see an Exception.

{code}
Class.forName("org.apache.drill.jdbc.Driver").newInstance();
Connection conn = DriverManager.getConnection(URL_STRING,"","");
Statement stmt = conn.createStatement();

String query = "select * from FEWRWSPQQ_101";
ResultSet rs = stmt.executeQuery(query);
ResultSetMetaData rsmd = rs.getMetaData();

while (rs.next()) {
}
conn.commit();

if (rs != null)
rs.close();
stmt.close();
conn.close();
{code}

Here is the Exception I see when I compile and run the Java program

{code}
Can't call commit() in auto-commit mode.
org.apache.drill.jdbc.JdbcApiSqlException: Can't call commit() in auto-commit 
mode.
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.commit(DrillConnectionImpl.java:192)
at DataFromDrill.main(DataFromDrill.java:30)
{code}


> JDBC : Connection.commit throws an UnsupportedOperationException
> 
>
> Key: DRILL-2451
> URL: https://issues.apache.org/jira/browse/DRILL-2451
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
> Fix For: 1.2.0
>
>
> git.commit.id.abbrev=e92db23
> Currently drill throws an UnsupportedOperationException when we call "commit" 
> on the Connection object. 
> I am not exactly sure what "commit" should do in the context of drill. But at 
> the very least doing nothing is better than throwing the above exception 
> since a few analytic tools might be using this method.
> Below is the documentation from the JDBC spec :
> {code}
> void commit() throws SQLException - Makes all changes made since the previous 
> commit/rollback permanent and releases any database locks currently held by 
> this Connection object. This method should be used only when auto-commit mode 
> has been disabled.
> Throws:
> SQLException - if a database access error occurs, this method is called while 
> participating in a distributed transaction, if this method is called on a 
> closed connection or this Connection object is in auto-commit mode
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2451) JDBC : Connection.commit throws an UnsupportedOperationException

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804862#comment-14804862
 ] 

Khurram Faraaz commented on DRILL-2451:
---

I see this when I try to turn off auto commit like this,
conn.setAutoCommit(false);

{code}
java.sql.SQLFeatureNotSupportedException: Can't turn off auto-committing; 
transactions are not supported.  (Drill is not transactional.)
at 
org.apache.drill.jdbc.impl.DrillConnectionImpl.setAutoCommit(DrillConnectionImpl.java:181)
at DataFromDrill.main(DataFromDrill.java:24)
{code}

> JDBC : Connection.commit throws an UnsupportedOperationException
> 
>
> Key: DRILL-2451
> URL: https://issues.apache.org/jira/browse/DRILL-2451
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
> Fix For: 1.2.0
>
>
> git.commit.id.abbrev=e92db23
> Currently drill throws an UnsupportedOperationException when we call "commit" 
> on the Connection object. 
> I am not exactly sure what "commit" should do in the context of drill. But at 
> the very least doing nothing is better than throwing the above exception 
> since a few analytic tools might be using this method.
> Below is the documentation from the JDBC spec :
> {code}
> void commit() throws SQLException - Makes all changes made since the previous 
> commit/rollback permanent and releases any database locks currently held by 
> this Connection object. This method should be used only when auto-commit mode 
> has been disabled.
> Throws:
> SQLException - if a database access error occurs, this method is called while 
> participating in a distributed transaction, if this method is called on a 
> closed connection or this Connection object is in auto-commit mode
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (DRILL-3333) Add support for auto-partitioning in parquet writer

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz closed DRILL-.
-

Verified.

> Add support for auto-partitioning in parquet writer
> ---
>
> Key: DRILL-
> URL: https://issues.apache.org/jira/browse/DRILL-
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Steven Phillips
>Assignee: Steven Phillips
> Fix For: 1.1.0
>
> Attachments: DRILL-.patch, DRILL-.patch, 
> DRILL-_2015-06-22_15:22:11.patch, DRILL-_2015-06-23_17:38:32.patch, 
> DRILL-_tests.patch
>
>
> When a table is created with a partition by clause, the parquet writer will 
> create separate files for the different partition values. The data will first 
> be sorted by the partition keys, and the parquet writer will create new file 
> when it encounters a new value for the partition columns.
> When data is queried against the data that was created this way, partition 
> pruning will work if the filter contains a partition column. And unlike 
> directory based partitioning, no view is required, nor is it necessary to 
> reference the dir* column names.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2451) JDBC : Connection.commit throws an UnsupportedOperationException

[Daniel Barclay (Drill) (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804886#comment-14804886
 ] 

Daniel Barclay (Drill) commented on DRILL-2451:
---

See the documentation (the Javadoc comment on DrillConnection.commit()).

> JDBC : Connection.commit throws an UnsupportedOperationException
> 
>
> Key: DRILL-2451
> URL: https://issues.apache.org/jira/browse/DRILL-2451
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Reporter: Rahul Challapalli
>Assignee: Rahul Challapalli
> Fix For: 1.2.0
>
>
> git.commit.id.abbrev=e92db23
> Currently drill throws an UnsupportedOperationException when we call "commit" 
> on the Connection object. 
> I am not exactly sure what "commit" should do in the context of drill. But at 
> the very least doing nothing is better than throwing the above exception 
> since a few analytic tools might be using this method.
> Below is the documentation from the JDBC spec :
> {code}
> void commit() throws SQLException - Makes all changes made since the previous 
> commit/rollback permanent and releases any database locks currently held by 
> this Connection object. This method should be used only when auto-commit mode 
> has been disabled.
> Throws:
> SQLException - if a database access error occurs, this method is called while 
> participating in a distributed transaction, if this method is called on a 
> closed connection or this Connection object is in auto-commit mode
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3776) count(*) from empty text file does not return 0

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu updated DRILL-3776:
-
Priority: Major  (was: Critical)

> count(*) from empty text file does not return 0
> ---
>
> Key: DRILL-3776
> URL: https://issues.apache.org/jira/browse/DRILL-3776
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>
> {code}
> select count(*) from `empty.csv`;
> {code}
> {code}
> select count(*) from `empty.json`;
> {code}
> Both return no record, no schema. But in postgres, it returns 0 for both 
> cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3676) Group by ordinal number of an output column results in parse error

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu updated DRILL-3676:
-
Priority: Major  (was: Critical)

> Group by ordinal number of an output column results in parse error
> --
>
> Key: DRILL-3676
> URL: https://issues.apache.org/jira/browse/DRILL-3676
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.4.0
>
>
> Group by number results in parse error.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_q.col1 from (select col1 from 
> FEWRWSPQQ_101) sub_q group by 1;
> Error: PARSE ERROR: At line 1, column 8: Expression 'q.col1' is not being 
> grouped
> [Error Id: 0eedafd9-372e-4610-b7a8-d97e26458d58 on centos-02.qa.lab:31010] 
> (state=,code=0)
> {code}
> When we use the column name instead of the number, the query compiles and 
> returns results.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col1 from (select col1 from 
> FEWRWSPQQ_101) group by col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1   |
> | 0|
> | 1|
> | 13   |
> | 17   |
> | 23   |
> | 1000 |
> | 999  |
> | 30   |
> | 25   |
> | 1001 |
> | -65535   |
> | 5000 |
> | 3000 |
> | 200  |
> | 197  |
> | 4611686018427387903  |
> | 9223372036854775806  |
> | 9223372036854775807  |
> | 92233720385475807|
> +--+
> 22 rows selected (0.218 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_query.col1 from (select col1 from 
> FEWRWSPQQ_101) sub_query group by sub_query.col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1   |
> | 0|
> | 1|
> | 13   |
> | 17   |
> | 23   |
> | 1000 |
> | 999  |
> | 30   |
> | 25   |
> | 1001 |
> | -65535   |
> | 5000 |
> | 3000 |
> | 200  |
> | 197  |
> | 4611686018427387903  |
> | 9223372036854775806  |
> | 9223372036854775807  |
> | 92233720385475807|
> +--+
> 22 rows selected (0.177 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman updated DRILL-3238:

Fix Version/s: (was: 1.2.0)
   1.3.0

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>  Labels: window_function
> Fix For: 1.3.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-1942) Improve off-heap memory usage tracking

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804694#comment-14804694
 ] 

ASF GitHub Bot commented on DRILL-1942:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/154


> Improve off-heap memory usage tracking
> --
>
> Key: DRILL-1942
> URL: https://issues.apache.org/jira/browse/DRILL-1942
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Reporter: Chris Westin
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: DRILL-1942.1.patch.txt, DRILL-1942.2.patch.txt, 
> DRILL-1942.3.patch.txt
>
>
> We're using a lot more memory than we think we should. We may be leaking it, 
> or not releasing it as soon as we could. 
> This is a call to come up with some improved tracking so that we can get 
> statistics out about exactly where we're using it, and whether or not we can 
> release it earlier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-3798) Cannot group by the functions without ()

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni reassigned DRILL-3798:
-

Assignee: Jinfeng Ni  (was: Mehant Baid)

> Cannot group by the functions without ()
> 
>
> Key: DRILL-3798
> URL: https://issues.apache.org/jira/browse/DRILL-3798
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Jinfeng Ni
>
> Drill can not group-by the function without ().
> eg:
> {code}
> SELECT CURRENT_DATE 
> FROM hive.h1db.testdate
> group  by CURRENT_DATE;
>   Caused By (org.apache.calcite.sql.validate.SqlValidatorException) Column 
> 'CURRENT_DATE' not found in any table
> {code}
> Bad ones:
> {code}
> SELECT CURRENT_TIME 
> FROM hive.h1db.testdate
> group  by CURRENT_TIME;
> SELECT CURRENT_TIMESTAMP 
> FROM hive.h1db.testdate
> group  by CURRENT_TIMESTAMP;
> SELECT LOCALTIME 
> FROM hive.h1db.testdate
> group  by LOCALTIME;
> SELECT LOCALTIMESTAMP 
> FROM hive.h1db.testdate
> group  by LOCALTIMESTAMP;
> {code}
> Good ones:
> {code}
> SELECT NOW()
> FROM hive.h1db.testdate
> group  by NOW();
> SELECT TIMEOFDAY()
> FROM hive.h1db.testdate
> group  by TIMEOFDAY();
> SELECT UNIX_TIMESTAMP()
> FROM hive.h1db.testdate
> group  by UNIX_TIMESTAMP();
> SELECT PI()
> FROM hive.h1db.testdate
> group  by  PI();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804671#comment-14804671
 ] 

Victoria Markman commented on DRILL-3238:
-

Lowering priority and moving to 1.3.0 - rare grammar.

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman updated DRILL-3238:

Priority: Major  (was: Critical)

> Cannot Plan Exception is raised when the same window partition is defined in 
> select & window clauses
> 
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>  Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by 
> a2 order by a2) 
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2) 
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the 
> window definition back into the SELECT-CLAUSE in the 2nd query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3676) Group by ordinal number of an output column results in parse error

[Sean Hsuan-Yi Chu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804672#comment-14804672
 ] 

Sean Hsuan-Yi Chu commented on DRILL-3676:
--

Lower the priority because it is more like a new (more convenient certainly) 
way to refer things. 

> Group by ordinal number of an output column results in parse error
> --
>
> Key: DRILL-3676
> URL: https://issues.apache.org/jira/browse/DRILL-3676
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.4.0
>
>
> Group by number results in parse error.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_q.col1 from (select col1 from 
> FEWRWSPQQ_101) sub_q group by 1;
> Error: PARSE ERROR: At line 1, column 8: Expression 'q.col1' is not being 
> grouped
> [Error Id: 0eedafd9-372e-4610-b7a8-d97e26458d58 on centos-02.qa.lab:31010] 
> (state=,code=0)
> {code}
> When we use the column name instead of the number, the query compiles and 
> returns results.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select col1 from (select col1 from 
> FEWRWSPQQ_101) group by col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1   |
> | 0|
> | 1|
> | 13   |
> | 17   |
> | 23   |
> | 1000 |
> | 999  |
> | 30   |
> | 25   |
> | 1001 |
> | -65535   |
> | 5000 |
> | 3000 |
> | 200  |
> | 197  |
> | 4611686018427387903  |
> | 9223372036854775806  |
> | 9223372036854775807  |
> | 92233720385475807|
> +--+
> 22 rows selected (0.218 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select sub_query.col1 from (select col1 from 
> FEWRWSPQQ_101) sub_query group by sub_query.col1;
> +--+
> | col1 |
> +--+
> | 65534|
> | 1000 |
> | -1   |
> | 0|
> | 1|
> | 13   |
> | 17   |
> | 23   |
> | 1000 |
> | 999  |
> | 30   |
> | 25   |
> | 1001 |
> | -65535   |
> | 5000 |
> | 3000 |
> | 200  |
> | 197  |
> | 4611686018427387903  |
> | 9223372036854775806  |
> | 9223372036854775807  |
> | 92233720385475807|
> +--+
> 22 rows selected (0.177 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3779) NPE during mergeAndSpill operation of external sort

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804737#comment-14804737
 ] 

ASF GitHub Bot commented on DRILL-3779:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/160


> NPE during mergeAndSpill operation of external sort
> ---
>
> Key: DRILL-3779
> URL: https://issues.apache.org/jira/browse/DRILL-3779
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Aman Sinha
>Assignee: Aman Sinha
> Fix For: 1.2.0
>
>
> For a query of the following pattern: 
> {code}
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.width.max_per_node` = 6;
> select 
>t1.c1, t1.c2t1.c50 
>   from t1 left outer join t2 on ..
>left outer join t3  on ...
>left outer join t4 on ...
>left outer join t5 on ...
> where t2.col between t1.c1 and t1.c2
> {code}
> we get an NPE below.  The dataset is large so I cannot attach it here.  Will 
> try to get a repro with smaller data.  
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 5:0
> [Error Id: 3df8da3d-dee2-45d4-9a0a-abaa0ab3775e on se-node11.se.lab:31010]
>   (java.lang.NullPointerException) null
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.getBufferSize():523
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():461
> 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():334
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():92
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft():127
> org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial():143
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext():171
> org.apache.drill.exec.record.AbstractRecordBatch.next():147
> org.apache.drill.exec.record.AbstractRecordBatch.next():105
> org.apache.drill.exec.record.AbstractRecordBatch.next():95
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-3798) Cannot group by the functions without ()

[Hao Zhu (JIRA)]

Hao Zhu created DRILL-3798:
--

 Summary: Cannot group by the functions without ()
 Key: DRILL-3798
 URL: https://issues.apache.org/jira/browse/DRILL-3798
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Drill
Affects Versions: 1.1.0
Reporter: Hao Zhu
Assignee: Mehant Baid


Drill can not group-by the function without ().
eg:
{code}
SELECT CURRENT_DATE 
FROM hive.h1db.testdate
group  by CURRENT_DATE;


  Caused By (org.apache.calcite.sql.validate.SqlValidatorException) Column 
'CURRENT_DATE' not found in any table
{code}

Bad ones:
{code}
SELECT CURRENT_TIME 
FROM hive.h1db.testdate
group  by CURRENT_TIME;

SELECT CURRENT_TIMESTAMP 
FROM hive.h1db.testdate
group  by CURRENT_TIMESTAMP;

SELECT LOCALTIME 
FROM hive.h1db.testdate
group  by LOCALTIME;

SELECT LOCALTIMESTAMP 
FROM hive.h1db.testdate
group  by LOCALTIMESTAMP;
{code}

Good ones:
{code}
SELECT NOW()
FROM hive.h1db.testdate
group  by NOW();

SELECT TIMEOFDAY()
FROM hive.h1db.testdate
group  by TIMEOFDAY();

SELECT UNIX_TIMESTAMP()
FROM hive.h1db.testdate
group  by UNIX_TIMESTAMP();

SELECT PI()
FROM hive.h1db.testdate
group  by  PI();
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-2861) enhance drill profile file management

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-2861:
---
Fix Version/s: (was: 1.2.0)
   Future

> enhance drill profile file management
> -
>
> Key: DRILL-2861
> URL: https://issues.apache.org/jira/browse/DRILL-2861
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Other
>Affects Versions: 0.9.0
>Reporter: Chun Chang
>Assignee: Sudheesh Katkam
> Fix For: Future
>
>
> We need to manage profile files better. Currently each query creates one 
> profile file on the local filesystem of the forman node. You can imagine how 
> this can quickly get out of hand in a production environment.
> We need:
> 1. be able to turn on and off profiling, preferably on the fly
> 2. profiling files should be managed the same as log files
> 3. able to change default file location, for example on a distributed 
> filesystem



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2583) Querying a non-existent table from hbase should throw a proper error message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804709#comment-14804709
 ] 

ASF GitHub Bot commented on DRILL-2583:
---

GitHub user sudheeshkatkam opened a pull request:

https://github.com/apache/drill/pull/161

DRILL-2583, DRILL-3428: Catch exceptions, and throw UserException#dat…

…aReadError with more context

+ Added convenient method to UserException for String.format(...)

@parthchandra please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sudheeshkatkam/drill DRILL-2583

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/161.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #161


commit 247b30160fe7a5da0a2b3dae3f46749085188c53
Author: Sudheesh Katkam 
Date:   2015-09-17T23:48:31Z

DRILL-2583, DRILL-3428: Catch exceptions, and throw 
UserException#dataReadError with more context

+ Added convenient method to UserException for String.format(...)




> Querying a non-existent table from hbase should throw a proper error message
> 
>
> Key: DRILL-2583
> URL: https://issues.apache.org/jira/browse/DRILL-2583
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: 0.8.0
>Reporter: Rahul Challapalli
>Assignee: Sudheesh Katkam
>  Labels: error_message_must_fix
> Fix For: 1.2.0
>
>
> git.commit.id.abbrev=4d398ed
> In the below case we should throw a proper error message
> {code}
> select count(*) from hbase.`non_existent_table`;
> Query failed: NullPointerException: 
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {code}
> We get the below error from DFS
> {code}
> Query failed: SqlValidatorException: Table 'dfs.fewtypes_null' not found
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)