[jira] [Updated] (DRILL-4678) Tune metadata by generating a dispatcher at runtime

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4678:
-
Summary: Tune metadata by generating a dispatcher at runtime  (was: Query 
HANG - SELECT DISTINCT over date data)

> Tune metadata by generating a dispatcher at runtime
> ---
>
> Key: DRILL-4678
> URL: https://issues.apache.org/jira/browse/DRILL-4678
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Attachments: hung_Date_Query.log
>
>
> Below query hangs
> {noformat}
> 2016-05-16 10:33:57,506 [28c65de9-9f67-dadb-5e4e-e1a12f8dda49:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 28c65de9-9f67-dadb-5e4e-e1a12f8dda49: SELECT DISTINCT dt FROM (
> VALUES(CAST('1964-03-07' AS DATE)),
>   (CAST('2002-03-04' AS DATE)),
>   (CAST('1966-09-04' AS DATE)),
>   (CAST('1993-08-18' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1959-10-23' AS DATE)),
>   (CAST('1992-01-14' AS DATE)),
>   (CAST('1994-07-24' AS DATE)),
>   (CAST('1979-11-25' AS DATE)),
>   (CAST('1945-01-14' AS DATE)),
>   (CAST('1982-07-25' AS DATE)),
>   (CAST('1966-09-06' AS DATE)),
>   (CAST('1989-05-01' AS DATE)),
>   (CAST('1996-03-08' AS DATE)),
>   (CAST('1998-08-19' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
> (CAST('1999-07-20' AS DATE)),
> (CAST('1962-07-03' AS DATE)),
>   (CAST('2011-08-17' AS DATE)),
>   (CAST('2011-05-16' AS DATE)),
>   (CAST('1946-05-08' AS DATE)),
>   (CAST('1994-02-13' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1958-02-06' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('1998-03-26' AS DATE)),
>   (CAST('1996-11-04' AS DATE)),
>   (CAST('1953-09-25' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('1980-07-05' AS DATE)),
>   (CAST('1982-06-15' AS DATE)),
>   (CAST('1951-05-16' AS DATE)))
> tbl(dt)
> {noformat}
> Details from Web UI Profile tab, please note that the query is still in 
> STARTING state
> {noformat}
> Running Queries
> Time  UserQuery   State   Foreman
> 05/16/2016 10:33:57   
> mapr
>  SELECT DISTINCT dt FROM ( VALUES(CAST('1964-03-07' AS DATE)), 
> (CAST('2002-03-04' AS DATE)), (CAST('1966-09-04' AS DATE)), (CAST('199
> STARTING
> centos-01.qa.lab
> {noformat}
> There is no other useful information in drillbit.log. jstack output is 
> attached here for your reference.
> The same query works fine on Postgres 9.3



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (DRILL-4678) Query HANG - SELECT DISTINCT over date data

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4678:


Assignee: Serhii Harnyk

> Query HANG - SELECT DISTINCT over date data
> ---
>
> Key: DRILL-4678
> URL: https://issues.apache.org/jira/browse/DRILL-4678
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.7.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Attachments: hung_Date_Query.log
>
>
> Below query hangs
> {noformat}
> 2016-05-16 10:33:57,506 [28c65de9-9f67-dadb-5e4e-e1a12f8dda49:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 28c65de9-9f67-dadb-5e4e-e1a12f8dda49: SELECT DISTINCT dt FROM (
> VALUES(CAST('1964-03-07' AS DATE)),
>   (CAST('2002-03-04' AS DATE)),
>   (CAST('1966-09-04' AS DATE)),
>   (CAST('1993-08-18' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1970-06-11' AS DATE)),
>   (CAST('1959-10-23' AS DATE)),
>   (CAST('1992-01-14' AS DATE)),
>   (CAST('1994-07-24' AS DATE)),
>   (CAST('1979-11-25' AS DATE)),
>   (CAST('1945-01-14' AS DATE)),
>   (CAST('1982-07-25' AS DATE)),
>   (CAST('1966-09-06' AS DATE)),
>   (CAST('1989-05-01' AS DATE)),
>   (CAST('1996-03-08' AS DATE)),
>   (CAST('1998-08-19' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
>   (CAST('2013-08-13' AS DATE)),
> (CAST('1999-07-20' AS DATE)),
> (CAST('1962-07-03' AS DATE)),
>   (CAST('2011-08-17' AS DATE)),
>   (CAST('2011-05-16' AS DATE)),
>   (CAST('1946-05-08' AS DATE)),
>   (CAST('1994-02-13' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1978-08-09' AS DATE)),
>   (CAST('1958-02-06' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('2012-06-11' AS DATE)),
>   (CAST('1998-03-26' AS DATE)),
>   (CAST('1996-11-04' AS DATE)),
>   (CAST('1953-09-25' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('2003-06-17' AS DATE)),
>   (CAST('1980-07-05' AS DATE)),
>   (CAST('1982-06-15' AS DATE)),
>   (CAST('1951-05-16' AS DATE)))
> tbl(dt)
> {noformat}
> Details from Web UI Profile tab, please note that the query is still in 
> STARTING state
> {noformat}
> Running Queries
> Time  UserQuery   State   Foreman
> 05/16/2016 10:33:57   
> mapr
>  SELECT DISTINCT dt FROM ( VALUES(CAST('1964-03-07' AS DATE)), 
> (CAST('2002-03-04' AS DATE)), (CAST('1966-09-04' AS DATE)), (CAST('199
> STARTING
> centos-01.qa.lab
> {noformat}
> There is no other useful information in drillbit.log. jstack output is 
> attached here for your reference.
> The same query works fine on Postgres 9.3



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Labels:   (was: ready-to-commit)

> Prevent left NLJoin with non scalar subqueries
> --
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join and non scalar sub-queries. Drill 
> should throw error in this case. 
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-4842) SELECT * on JSON data results in NumberFormatException

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4842:
-
Labels:   (was: ready-to-commit)

> SELECT * on JSON data results in NumberFormatException
> --
>
> Key: DRILL-4842
> URL: https://issues.apache.org/jira/browse/DRILL-4842
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Attachments: tooManyNulls.json
>
>
> Note that doing SELECT c1 returns correct results, the failure is seen when 
> we do SELECT star. json.all_text_mode was set to true.
> JSON file tooManyNulls.json has one key c1 with 4096 nulls as its value and 
> the 4097th key c1 has the value "Hello World"
> git commit ID : aaf220ff
> MapR Drill 1.8.0 RPM
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.27 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT c1 FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> +--+
> |  c1  |
> +--+
> | Hello World  |
> +--+
> 1 row selected (0.243 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> Error: SYSTEM ERROR: NumberFormatException: Hello World
> Fragment 0:0
> [Error Id: 9cafb3f9-3d5c-478a-b55c-900602b8765e on centos-01.qa.lab:31010]
>  (java.lang.NumberFormatException) Hello World
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI():95
> 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt():120
> org.apache.drill.exec.test.generated.FiltererGen1169.doSetup():45
> org.apache.drill.exec.test.generated.FiltererGen1169.setup():54
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer():195
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema():107
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745 (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.NumberFormatException: Hello World
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI(StringFunctionHelpers.java:95)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt(StringFunctionHelpers.java:120)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> 

[jira] [Updated] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Labels: ready-to-commit  (was: )

> Prevent left NLJoin with non scalar subqueries
> --
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join and non scalar sub-queries. Drill 
> should throw error in this case. 
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4939) to_date function returns incorrect result

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869573#comment-15869573
 ] 

Serhii Harnyk commented on DRILL-4939:
--

[~khfaraaz] This function returns correct result, and it's behavior does not 
changed. According to 
http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html
 you should use pattern -MM-dd. If you want to use Postgres-based date-time 
format, you should use function sql_to_date.
{code:sql}
values(to_date('2016-09-22','-MM-dd'));
+-+
|   EXPR$0|
+-+
| 2016-09-22  |
+-+
{code}

> to_date function returns incorrect result
> -
>
> Key: DRILL-4939
> URL: https://issues.apache.org/jira/browse/DRILL-4939
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Priority: Critical
>
> to_date function returns wrong result
> correct results from Postgres
> {noformat}
> postgres=# values(to_date('2016-09-22','-mm-dd'));
>   column1
> 
>  2016-09-22
> (1 row)
> {noformat}
> wrong results returned by Drill 1.9.0 git commit id: 4edabe7a
> {noformat}
> : jdbc:drill:schema=dfs.tmp> values(to_date('2016-09-22','-mm-dd'));
> +-+
> |   EXPR$0|
> +-+
> | 2016-01-22  |
> +-+
> 1 row selected (0.125 seconds)
> {noformat}
> Postgres 9.3 returns true for below query whereas drill 1.9.0 returns false.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 
> to_date('2016-09-22','-mm-dd')) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> ++
> |  res2  |
> ++
> | false  |
> ++
> 1 row selected (0.146 seconds)
> postgres=# select (res1 = to_date('2016/09/22','-mm-dd')) res2
> postgres-# from
> postgres-# (
> postgres(# select (case when (false) then null else cast('2016/09/22' as 
> date) end) res1
> postgres(# from (values(1)) foo
> postgres(# ) foobar;
>  res2
> --
>  t
> (1 row)
> {noformat}
> Postgres 9.3 returns an error for below query, where as Drill git commit ID: 
> 4edabe7a returns some results.
> This looks like it has to do with the to_date function in drill.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = to_date(2016/09/22)) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> ++
> |  res2  |
> ++
> | false  |
> ++
> 1 row selected (0.166 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Summary: Prevent left NLJoin with non scalar subqueries  (was: Prevent left 
nested loop join)

> Prevent left NLJoin with non scalar subqueries
> --
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join. Drill should throw error in this 
> case.
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Description: 
Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join and non scalar sub-queries. Drill 
should throw error in this case. 
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}

  was:
Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join. Drill should throw error in this 
case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}


> Prevent left NLJoin with non scalar subqueries
> --
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join and non scalar sub-queries. Drill 
> should throw error in this case. 
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5263) Prevent left nested loop join

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868199#comment-15868199
 ] 

Serhii Harnyk commented on DRILL-5263:
--

Before this changes allowed only left and inner NLJoins.

> Prevent left nested loop join
> -
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join. Drill should throw error in this 
> case.
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left nested loop join

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Description: 
Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join. Drill should throw error in this 
case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}

  was:
Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join and non scalar sub-queries. Drill 
should throw error in this case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}


> Prevent left nested loop join
> -
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join. Drill should throw error in this 
> case.
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left nested loop join

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Summary: Prevent left nested loop join  (was: Prevent left NLJoin with non 
scalar subqueries)

> Prevent left nested loop join
> -
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join and non scalar sub-queries. Drill 
> should throw error in this case.
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5263:
-
Attachment: tmp.tar.gz

> Prevent left NLJoin with non scalar subqueries
> --
>
> Key: DRILL-5263
> URL: https://issues.apache.org/jira/browse/DRILL-5263
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: tmp.tar.gz
>
>
> Nested loop join operator in Drill supports only inner join and returns 
> incorrect result for queries with left join and non scalar sub-queries. Drill 
> should throw error in this case.
> Example:
> {code:sql}
> alter session set planner.enable_nljoin_for_scalar_only=false;
> select t2.dt, t1.fyq, t2.who, t2.event
> from t2
> left join t1 on t2.dt between t1.dts and t1.dte
> order by t2.dt;
> {code}
> Result:
> {noformat}
> +-+--+--++
> | dt  |   fyq|   who|   event|
> +-+--+--++
> | 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
> | 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
> | 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
> +-+--+--++
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5263) Prevent left NLJoin with non scalar subqueries

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5263:


 Summary: Prevent left NLJoin with non scalar subqueries
 Key: DRILL-5263
 URL: https://issues.apache.org/jira/browse/DRILL-5263
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Nested loop join operator in Drill supports only inner join and returns 
incorrect result for queries with left join and non scalar sub-queries. Drill 
should throw error in this case.
Example:
{code:sql}
alter session set planner.enable_nljoin_for_scalar_only=false;
select t2.dt, t1.fyq, t2.who, t2.event
from t2
left join t1 on t2.dt between t1.dts and t1.dte
order by t2.dt;
{code}
Result:
{noformat}
+-+--+--++
| dt  |   fyq|   who|   event|
+-+--+--++
| 2016-12-26  | 2016-Q2  | aperson  | had chrsitmas  |
| 2017-01-06  | 2016-Q3  | aperson  | did somthing   |
| 2017-01-12  | 2016-Q3  | aperson  | did somthing else  |
+-+--+--++
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4864) Add ANSI format for date/time functions

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863355#comment-15863355
 ] 

Serhii Harnyk commented on DRILL-4864:
--

Fixed in 
[c9a6ac4|https://github.com/apache/drill/commit/c9a6ac4fc8859693b7b0a71885afb700f81e345c]

> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.10
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global option (see usage notes)|fx |
> |translation mode (print localized day and month names based on 
> lc_messages)|tm |
> |spell mode (not yet implemented)|sp|
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-4939) to_date function returns incorrect result

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863390#comment-15863390
 ] 

Serhii Harnyk commented on DRILL-4939:
--

to_date function requires date-time pattern in JodaTime format 
http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html
where mm is a minute of hour. 
In 
[c9a6ac4|https://github.com/apache/drill/commit/c9a6ac4fc8859693b7b0a71885afb700f81e345c]
 added new UDF sql_to_date, which requires Postgres-based date-time format.
So next query returns correct result:
{code} 
0: jdbc:drill:zk=local> values(sql_to_date('2016-09-22','-mm-dd'));
+-+
|   EXPR$0|
+-+
| 2016-09-22  |
+-+
{code}

> to_date function returns incorrect result
> -
>
> Key: DRILL-4939
> URL: https://issues.apache.org/jira/browse/DRILL-4939
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Priority: Critical
>
> to_date function returns wrong result
> correct results from Postgres
> {noformat}
> postgres=# values(to_date('2016-09-22','-mm-dd'));
>   column1
> 
>  2016-09-22
> (1 row)
> {noformat}
> wrong results returned by Drill 1.9.0 git commit id: 4edabe7a
> {noformat}
> : jdbc:drill:schema=dfs.tmp> values(to_date('2016-09-22','-mm-dd'));
> +-+
> |   EXPR$0|
> +-+
> | 2016-01-22  |
> +-+
> 1 row selected (0.125 seconds)
> {noformat}
> Postgres 9.3 returns true for below query whereas drill 1.9.0 returns false.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 
> to_date('2016-09-22','-mm-dd')) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> ++
> |  res2  |
> ++
> | false  |
> ++
> 1 row selected (0.146 seconds)
> postgres=# select (res1 = to_date('2016/09/22','-mm-dd')) res2
> postgres-# from
> postgres-# (
> postgres(# select (case when (false) then null else cast('2016/09/22' as 
> date) end) res1
> postgres(# from (values(1)) foo
> postgres(# ) foobar;
>  res2
> --
>  t
> (1 row)
> {noformat}
> Postgres 9.3 returns an error for below query, where as Drill git commit ID: 
> 4edabe7a returns some results.
> This looks like it has to do with the to_date function in drill.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = to_date(2016/09/22)) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> ++
> |  res2  |
> ++
> | false  |
> ++
> 1 row selected (0.166 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-4864) Add ANSI format for date/time functions

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4864:
-
Labels: doc-impacting  (was: doc-impacting ready-to-commit)

> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: 1.10
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|am
> |Year   |   y   
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |
> Table of acceptable Postgres pattern modifiers, which may be used in Format 
> string
> ||Description||Pattern||
> |fill mode (suppress padding blanks and zeroes)|fm |
> |fixed format global option (see usage notes)|fx |
> |translation mode (print localized day and month names based on 
> lc_messages)|tm |
> |spell mode (not yet implemented)|sp|
> |ref.|
> https://www.postgresql.org/docs/8.2/static/functions-formatting.html|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-4842) SELECT * on JSON data results in NumberFormatException

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4842:
-
Labels:   (was: ready-to-commit)

> SELECT * on JSON data results in NumberFormatException
> --
>
> Key: DRILL-4842
> URL: https://issues.apache.org/jira/browse/DRILL-4842
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Attachments: tooManyNulls.json
>
>
> Note that doing SELECT c1 returns correct results, the failure is seen when 
> we do SELECT star. json.all_text_mode was set to true.
> JSON file tooManyNulls.json has one key c1 with 4096 nulls as its value and 
> the 4097th key c1 has the value "Hello World"
> git commit ID : aaf220ff
> MapR Drill 1.8.0 RPM
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.27 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT c1 FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> +--+
> |  c1  |
> +--+
> | Hello World  |
> +--+
> 1 row selected (0.243 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> Error: SYSTEM ERROR: NumberFormatException: Hello World
> Fragment 0:0
> [Error Id: 9cafb3f9-3d5c-478a-b55c-900602b8765e on centos-01.qa.lab:31010]
>  (java.lang.NumberFormatException) Hello World
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI():95
> 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt():120
> org.apache.drill.exec.test.generated.FiltererGen1169.doSetup():45
> org.apache.drill.exec.test.generated.FiltererGen1169.setup():54
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer():195
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema():107
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745 (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.NumberFormatException: Hello World
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI(StringFunctionHelpers.java:95)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt(StringFunctionHelpers.java:120)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> 

[jira] [Updated] (DRILL-4842) SELECT * on JSON data results in NumberFormatException

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4842:
-
Labels: ready-to-commit  (was: )

> SELECT * on JSON data results in NumberFormatException
> --
>
> Key: DRILL-4842
> URL: https://issues.apache.org/jira/browse/DRILL-4842
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Attachments: tooManyNulls.json
>
>
> Note that doing SELECT c1 returns correct results, the failure is seen when 
> we do SELECT star. json.all_text_mode was set to true.
> JSON file tooManyNulls.json has one key c1 with 4096 nulls as its value and 
> the 4097th key c1 has the value "Hello World"
> git commit ID : aaf220ff
> MapR Drill 1.8.0 RPM
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.27 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT c1 FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> +--+
> |  c1  |
> +--+
> | Hello World  |
> +--+
> 1 row selected (0.243 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> Error: SYSTEM ERROR: NumberFormatException: Hello World
> Fragment 0:0
> [Error Id: 9cafb3f9-3d5c-478a-b55c-900602b8765e on centos-01.qa.lab:31010]
>  (java.lang.NumberFormatException) Hello World
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI():95
> 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt():120
> org.apache.drill.exec.test.generated.FiltererGen1169.doSetup():45
> org.apache.drill.exec.test.generated.FiltererGen1169.setup():54
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer():195
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema():107
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745 (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.NumberFormatException: Hello World
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI(StringFunctionHelpers.java:95)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt(StringFunctionHelpers.java:120)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>

[jira] [Commented] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches for the first time

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853676#comment-15853676
 ] 

Serhii Harnyk commented on DRILL-5237:
--

Fixed in 
[c8fbc38|https://github.com/apache/drill/commit/c8fbc386c93484b05307f77efe88333746cb8a5e]

> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches for the first time
> 
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches for the first time

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5237:
-
Summary: FlattenRecordBatch loses nested fields from the schema when 
returns empty batches for the first time  (was: FlattenRecordBatch loses nested 
fields from the schema when returns empty batches at first time)

> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches for the first time
> 
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches for the first time

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848337#comment-15848337
 ] 

Serhii Harnyk edited comment on DRILL-5237 at 2/1/17 12:56 PM:
---

This bug caused that FlattenRecordBatch loses nested fields from the schema 
when returns empty batches for the first time. When outer batches use these 
nested fields, they generate classes which operate with such fields like with 
constants. When next time FlattenRecordBatch returns non empty batch, generated 
classes of outer batches stay the same, so they returns wrong result.


was (Author: sharnyk):
This bug caused that FlattenRecordBatch loses nested fields from the schema 
when returns empty batches at first time. When outer batches use these nested 
fields, they generate classes which operate with such fields like with 
constants. When next time FlattenRecordBatch returns non empty batch, generated 
classes of outer batches stay the same, so they returns wrong result.

> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches for the first time
> 
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches at first time

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848337#comment-15848337
 ] 

Serhii Harnyk commented on DRILL-5237:
--

This bug caused that FlattenRecordBatch loses nested fields from the schema 
when returns empty batches at first time. When outer batches use these nested 
fields, they generate classes which operate with such fields like with 
constants. When next time FlattenRecordBatch returns non empty batch, generated 
classes of outer batches stay the same, so they returns wrong result.

> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches at first time
> ---
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5237) FlattenRecordBatch loses nested fields from the schema when returns empty batches at first time

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5237:
-
Summary: FlattenRecordBatch loses nested fields from the schema when 
returns empty batches at first time  (was: Same query produces different/wrong 
results)

> FlattenRecordBatch loses nested fields from the schema when returns empty 
> batches at first time
> ---
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (DRILL-5237) Same query produces different/wrong results

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5237:
-
Attachment: test_data.tar.gz

Files for reproducing this issue

> Same query produces different/wrong results
> ---
>
> Key: DRILL-5237
> URL: https://issues.apache.org/jira/browse/DRILL-5237
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Attachments: test_data.tar.gz
>
>
> Query 
> {code:sql}
> select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as 
> b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 
> 'a1');
> {code}
> returns different results for different values for option 
> planner.width.max_per_node
> With options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 1;
> {code}
> query returns correct result:
> {noformat}
> +--+
> | col  |
> +--+
> | 3|
> +--+
> {noformat}
> but with options 
> {code:sql}
> alter session set `planner.slice_target` = 1;
> alter session set `planner.width.max_per_node` = 3;
> {code}
> the same query returns wrong result 
> {noformat}
> +--+
> | col  |
> +--+
> | 2|
> +--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5237) Same query produces different/wrong results

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5237:


 Summary: Same query produces different/wrong results
 Key: DRILL-5237
 URL: https://issues.apache.org/jira/browse/DRILL-5237
 Project: Apache Drill
  Issue Type: Bug
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Query 
{code:sql}
select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as b 
from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 'a1');
{code}
returns different results for different values for option 
planner.width.max_per_node
With options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 1;
{code}
query returns correct result:
{noformat}
+--+
| col  |
+--+
| 3|
+--+
{noformat}
but with options 
{code:sql}
alter session set `planner.slice_target` = 1;
alter session set `planner.width.max_per_node` = 3;
{code}
the same query returns wrong result 
{noformat}
+--+
| col  |
+--+
| 2|
+--+
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] (DRILL-4578) "children" missing from results of full scan over JSON data

[Serhii Harnyk (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Serhii Harnyk resolved as Fixed 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
Fixed in 88655ad 
 
 
 
 
 
 
 
 
 
 Apache Drill /  DRILL-4578 
 
 
 
  "children" missing from results of full scan over JSON data  
 
 
 
 
 
 
 
 
 

Change By:
 
 Serhii Harnyk 
 
 
 

Resolution:
 
 Fixed 
 
 
 

Fix Version/s:
 
 1.10.0 
 
 
 

Status:
 
 Open Resolved 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   

[jira] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Serhii Harnyk updated an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Apache Drill /  DRILL-3562 
 
 
 
  Query fails when using flatten on JSON data where some documents have an empty array  
 
 
 
 
 
 
 
 
 

Change By:
 
 Serhii Harnyk 
 
 
 

Labels:
 
 ready-to-commit 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   

[jira] (DRILL-4764) Parquet file with INT_16, etc. logical types not supported by simple SELECT

[Serhii Harnyk (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Serhii Harnyk updated an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Apache Drill /  DRILL-4764 
 
 
 
  Parquet file with INT_16, etc. logical types not supported by simple SELECT  
 
 
 
 
 
 
 
 
 

Change By:
 
 Serhii Harnyk 
 
 
 

Labels:
 
 ready-to-commit 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   

[jira] (DRILL-4217) Query parquet file treat INT_16 & INT_8 as INT32

[Serhii Harnyk (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Serhii Harnyk resolved as Fixed 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
Fixed in 60624af 
 
 
 
 
 
 
 
 
 
 Apache Drill /  DRILL-4217 
 
 
 
  Query parquet file treat INT_16 & INT_8 as INT32  
 
 
 
 
 
 
 
 
 

Change By:
 
 Serhii Harnyk 
 
 
 

Resolution:
 
 Fixed 
 
 
 

Fix Version/s:
 
 1.10.0 
 
 
 

Status:
 
 In Progress Resolved 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   

[jira] [Updated] (DRILL-4824) JSON with complex nested data produces incorrect output with missing fields

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4824:
-
Reviewer: Paul Rogers  (was: Dechang Gu)

> JSON with complex nested data produces incorrect output with missing fields
> ---
>
> Key: DRILL-4824
> URL: https://issues.apache.org/jira/browse/DRILL-4824
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Roman
>Assignee: Serhii Harnyk
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
> "Field1" : {
> }
> }
> {
> "Field1" : {
> "InnerField1": {"key1":"value1"},
> "InnerField2": {"key2":"value2"}
> }
> }
> {
> "Field1" : {
> "InnerField3" : ["value3", "value4"],
> "InnerField4" : ["value5", "value6"]
> }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---+
> |  Field1   |
> +---+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" 
> {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}
> Theres is no need to output missing fields. In case of deeply nested 
> structure we will get unreadable result for user.
> _Correct result:_
> {code:none}
> +--+
> | Field1   |
> +--+
> |{} 
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4824) JSON with complex nested data produces incorrect output with missing fields

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838034#comment-15838034
 ] 

Serhii Harnyk commented on DRILL-4824:
--

Map field in json is a part of the schema, and it is clear that json has single 
schema for the all records. So we have such behaviour. 
[~Paul.Rogers], what is the expected result of select on such json file?
{noformat}
{
"Field1" : {
   "InnerField1": { }
}
}

{
"Field1" : {
"InnerField2": {"key2":"value2"}
}
}
{noformat}
In this case schema has fields Field1.InnerField1 and Field1.InnerField2.
Should the output for the first record be only { }?

> JSON with complex nested data produces incorrect output with missing fields
> ---
>
> Key: DRILL-4824
> URL: https://issues.apache.org/jira/browse/DRILL-4824
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Roman
>Assignee: Serhii Harnyk
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
> "Field1" : {
> }
> }
> {
> "Field1" : {
> "InnerField1": {"key1":"value1"},
> "InnerField2": {"key2":"value2"}
> }
> }
> {
> "Field1" : {
> "InnerField3" : ["value3", "value4"],
> "InnerField4" : ["value5", "value6"]
> }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---+
> |  Field1   |
> +---+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" 
> {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}
> Theres is no need to output missing fields. In case of deeply nested 
> structure we will get unreadable result for user.
> _Correct result:_
> {code:none}
> +--+
> | Field1   |
> +--+
> |{} 
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4824) JSON with complex nested data produces incorrect output with missing fields

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4824:
-
Fix Version/s: (was: 1.9.0)

> JSON with complex nested data produces incorrect output with missing fields
> ---
>
> Key: DRILL-4824
> URL: https://issues.apache.org/jira/browse/DRILL-4824
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Roman
>Assignee: Serhii Harnyk
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
> "Field1" : {
> }
> }
> {
> "Field1" : {
> "InnerField1": {"key1":"value1"},
> "InnerField2": {"key2":"value2"}
> }
> }
> {
> "Field1" : {
> "InnerField3" : ["value3", "value4"],
> "InnerField4" : ["value5", "value6"]
> }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---+
> |  Field1   |
> +---+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" 
> {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}
> Theres is no need to output missing fields. In case of deeply nested 
> structure we will get unreadable result for user.
> _Correct result:_
> {code:none}
> +--+
> | Field1   |
> +--+
> |{} 
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4824) JSON with complex nested data produces incorrect output with missing fields

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4824:


Assignee: Serhii Harnyk  (was: Roman)

> JSON with complex nested data produces incorrect output with missing fields
> ---
>
> Key: DRILL-4824
> URL: https://issues.apache.org/jira/browse/DRILL-4824
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Roman
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> There is incorrect output in case of JSON file with complex nested data.
> _JSON:_
> {code:none|title=example.json|borderStyle=solid}
> {
> "Field1" : {
> }
> }
> {
> "Field1" : {
> "InnerField1": {"key1":"value1"},
> "InnerField2": {"key2":"value2"}
> }
> }
> {
> "Field1" : {
> "InnerField3" : ["value3", "value4"],
> "InnerField4" : ["value5", "value6"]
> }
> }
> {code}
> _Query:_
> {code:sql}
> select Field1 from dfs.`/tmp/example.json`
> {code}
> _Incorrect result:_
> {code:none}
> +---+
> |  Field1   |
> +---+
> {"InnerField1":{},"InnerField2":{},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{"key1":"value1"},"InnerField2" 
> {"key2":"value2"},"InnerField3":[],"InnerField4":[]}
> {"InnerField1":{},"InnerField2":{},"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}
> Theres is no need to output missing fields. In case of deeply nested 
> structure we will get unreadable result for user.
> _Correct result:_
> {code:none}
> +--+
> | Field1   |
> +--+
> |{} 
> {"InnerField1":{"key1":"value1"},"InnerField2":{"key2":"value2"}}
> {"InnerField3":["value3","value4"],"InnerField4":["value5","value6"]}
> +--+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-3562:
-
Labels: ready-to-commit  (was: )

> Query fails when using flatten on JSON data where some documents have an 
> empty array
> 
>
> Key: DRILL-3562
> URL: https://issues.apache.org/jira/browse/DRILL-3562
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.1.0
>Reporter: Philip Deegan
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Fix For: Future
>
>
> Drill query fails when using flatten when some records contain an empty array 
> {noformat}
> SELECT COUNT(*) FROM (SELECT FLATTEN(t.a.b.c) AS c FROM dfs.`flat.json` t) 
> flat WHERE flat.c.d.e = 'f' limit 1;
> {noformat}
> Succeeds on 
> { "a": { "b": { "c": [  { "d": {  "e": "f" } } ] } } }
> Fails on
> { "a": { "b": { "c": [] } } }
> Error
> {noformat}
> Error: SYSTEM ERROR: ClassCastException: Cannot cast 
> org.apache.drill.exec.vector.NullableIntVector to 
> org.apache.drill.exec.vector.complex.RepeatedValueVector
> {noformat}
> Is it possible to ignore the empty arrays, or do they need to be populated 
> with dummy data?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5197) CASE statement fails due to error: Unable to get value vector class for minor type [NULL] and mode [OPTIONAL]

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825724#comment-15825724
 ] 

Serhii Harnyk commented on DRILL-5197:
--

[~mgelbana], No, Drill has such behavior according to ANSI standards. Casting 
is a good way to escape this problem.

> CASE statement fails due to error: Unable to get value vector class for minor 
> type [NULL] and mode [OPTIONAL]
> -
>
> Key: DRILL-5197
> URL: https://issues.apache.org/jira/browse/DRILL-5197
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Muhammad Gelbana
>
> The following query fails for no obvious reason
> {code:sql}
> SELECT
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN
>  ( 
>  CASE
> WHEN `tname`.`full_name` = 'ABC' 
> THEN
>(
>   CASE
>  WHEN `tname`.`full_name` = ' ' 
>  THEN
> (
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN `tname`.`full_name` 
>   ELSE NULL 
>END
> )
> ELSE NULL 
>   END
>)
>ELSE NULL 
>  END
>  )
>  WHEN `tname`.`full_name` = 'ABC' 
>  THEN NULL 
>  ELSE NULL 
>END
> FROM
>cp.`employee.json` `tname`
> {code}
> If the `THEN `tname`.`full_name`` statements is changed to `THEN 'ABC'` the 
> error does not occur.
> Thrown exception
> {noformat}
> [Error Id: e75fd0fe-132b-4eb4-b2e8-7b34dc39657e on mgelbana-incorta:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.9.0.jar:1.9.0]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> Caused by: java.lang.UnsupportedOperationException: Unable to get value 
> vector class for minor type [NULL] and mode [OPTIONAL]
>   at 
> org.apache.drill.exec.expr.BasicTypeHelper.getValueVectorClass(BasicTypeHelper.java:441)
>  ~[vector-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.VectorContainer.addOrGet(VectorContainer.java:123)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:463)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at java.security.AccessController.doPrivileged(Native Method) 
> ~[na:1.8.0_111]
>   at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_111]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>  ~[hadoop-common-2.7.1.jar:na]
>   at 
> 

[jira] [Commented] (DRILL-5197) CASE statement fails due to error: Unable to get value vector class for minor type [NULL] and mode [OPTIONAL]

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824162#comment-15824162
 ] 

Serhii Harnyk commented on DRILL-5197:
--

Replacing 
{noformat}
THEN `tname`.`full_name`
{noformat}
statement with a literal value you defines the type of result field as 
VARCHAR(REQUIRED), 
but the root of problem this part of query:
{noformat}
WHEN `tname`.`full_name` = 'ABC' 
 THEN NULL 
 ELSE NULL 
{noformat}
because drill couldn't define the type of result. 
You can ensure in that by replacing any "NULL" of this part of query.

> CASE statement fails due to error: Unable to get value vector class for minor 
> type [NULL] and mode [OPTIONAL]
> -
>
> Key: DRILL-5197
> URL: https://issues.apache.org/jira/browse/DRILL-5197
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Muhammad Gelbana
>
> The following query fails for no obvious reason
> {code:sql}
> SELECT
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN
>  ( 
>  CASE
> WHEN `tname`.`full_name` = 'ABC' 
> THEN
>(
>   CASE
>  WHEN `tname`.`full_name` = ' ' 
>  THEN
> (
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN `tname`.`full_name` 
>   ELSE NULL 
>END
> )
> ELSE NULL 
>   END
>)
>ELSE NULL 
>  END
>  )
>  WHEN `tname`.`full_name` = 'ABC' 
>  THEN NULL 
>  ELSE NULL 
>END
> FROM
>cp.`employee.json` `tname`
> {code}
> If the `THEN `tname`.`full_name`` statements is changed to `THEN 'ABC'` the 
> error does not occur.
> Thrown exception
> {noformat}
> [Error Id: e75fd0fe-132b-4eb4-b2e8-7b34dc39657e on mgelbana-incorta:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.9.0.jar:1.9.0]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> Caused by: java.lang.UnsupportedOperationException: Unable to get value 
> vector class for minor type [NULL] and mode [OPTIONAL]
>   at 
> org.apache.drill.exec.expr.BasicTypeHelper.getValueVectorClass(BasicTypeHelper.java:441)
>  ~[vector-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.VectorContainer.addOrGet(VectorContainer.java:123)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:463)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 

[jira] [Commented] (DRILL-5197) CASE statement fails due to error: Unable to get value vector class for minor type [NULL] and mode [OPTIONAL]

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823932#comment-15823932
 ] 

Serhii Harnyk commented on DRILL-5197:
--

[~khfaraaz], It is a duplicate of DRILL-4936. As mentioned there, according to 
the sql standard we can not have all of the THEN statements and the ELSE 
returning null.

> CASE statement fails due to error: Unable to get value vector class for minor 
> type [NULL] and mode [OPTIONAL]
> -
>
> Key: DRILL-5197
> URL: https://issues.apache.org/jira/browse/DRILL-5197
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Muhammad Gelbana
>
> The following query fails for no obvious reason
> {code:sql}
> SELECT
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN
> ( 
>  CASE
> WHEN `tname`.`full_name` = 'ABC' 
> THEN
>(
>   CASE
>  WHEN `tname`.`full_name` = ' ' 
>  THEN
> (
>CASE
>   WHEN `tname`.`full_name` = 'ABC' 
>   THEN `tname`.`full_name` 
>   ELSE NULL 
>END
> )
> ELSE NULL 
>   END
>)
>ELSE NULL 
>  END
>  )
>  WHEN `tname`.`full_name` = 'ABC' 
>  THEN NULL 
>  ELSE NULL 
>END
> FROM
>cp.`employee.json` `tname`
> {code}
> If the `THEN `tname`.`full_name`` statements is changed to `THEN 'ABC'` the 
> error does not occur.
> Thrown exception
> {noformat}
> [Error Id: e75fd0fe-132b-4eb4-b2e8-7b34dc39657e on mgelbana-incorta:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.9.0.jar:1.9.0]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_111]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_111]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> Caused by: java.lang.UnsupportedOperationException: Unable to get value 
> vector class for minor type [NULL] and mode [OPTIONAL]
>   at 
> org.apache.drill.exec.expr.BasicTypeHelper.getValueVectorClass(BasicTypeHelper.java:441)
>  ~[vector-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.VectorContainer.addOrGet(VectorContainer.java:123)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setupNewSchema(ProjectRecordBatch.java:463)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
>   at java.security.AccessController.doPrivileged(Native Method) 
> ~[na:1.8.0_111]
>   at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_111]
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>  

[jira] [Commented] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818118#comment-15818118
 ] 

Serhii Harnyk commented on DRILL-3562:
--

Besides the initialization of empty arrays we have the problem with ordering of 
columns with arrays. 
Query 
{code}
select * from example 
{code}
for Json
{noformat}
{ "a": [], "c": [], "c1": 1 }
{ "a": [1], "c": [1], "c1": 1 }
{noformat}
returns result
{noformat}
---
| c1| a | c |
---
| 1   | []  | []  |
| 1   | [1] | [1] |
---
{noformat}
with wrong columns order.

> Query fails when using flatten on JSON data where some documents have an 
> empty array
> 
>
> Key: DRILL-3562
> URL: https://issues.apache.org/jira/browse/DRILL-3562
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.1.0
>Reporter: Philip Deegan
>Assignee: Serhii Harnyk
> Fix For: Future
>
>
> Drill query fails when using flatten when some records contain an empty array 
> {noformat}
> SELECT COUNT(*) FROM (SELECT FLATTEN(t.a.b.c) AS c FROM dfs.`flat.json` t) 
> flat WHERE flat.c.d.e = 'f' limit 1;
> {noformat}
> Succeeds on 
> { "a": { "b": { "c": [  { "d": {  "e": "f" } } ] } } }
> Fails on
> { "a": { "b": { "c": [] } } }
> Error
> {noformat}
> Error: SYSTEM ERROR: ClassCastException: Cannot cast 
> org.apache.drill.exec.vector.NullableIntVector to 
> org.apache.drill.exec.vector.complex.RepeatedValueVector
> {noformat}
> Is it possible to ignore the empty arrays, or do they need to be populated 
> with dummy data?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4578) "children" missing from results of full scan over JSON data

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4578:


Assignee: Serhii Harnyk

> "children" missing from results of full scan over JSON data
> ---
>
> Key: DRILL-4578
> URL: https://issues.apache.org/jira/browse/DRILL-4578
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
>
> One of the fields named "children" is missing from the output of SELECT * 
> over the JSON data, with or without enabling all_text_mode for JSON data.
> Projecting just the "children" field returns a null.
> Note that children field holds an empty array.
>  Drill 1.7.0-SNAPSHOT  commit ID e7e9b73c
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.118 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * from `employee.json`;
> ++---+--+--++-+--+---+
> | firstName  | lastName  | isAlive  | age  | height_cm  | 
>   address   | 
> phoneNumbers  
>|hobbies|
> ++---+--+--++-+--+---+
> | John   | Smith | true | 45   | 177.6  | 
> {"streetAddress":"29 4th Street","city":"New 
> York","state":"NY","postalCode":"10021-3100"}  | 
> [{"type":"home","number":"212 555-1234"},{"type":"office","number":"646 
> 555-4567"}]  | ["scuba diving","hiking","biking","rock climbing","surfing"]  |
> ++---+--+--++-+--+---+
> 1 row selected (0.214 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select t.children from `employee.json` t;
> +---+
> | children  |
> +---+
> | null  |
> +---+
> 1 row selected (0.148 seconds)
> {noformat}
> JSON data used in test
> {noformat}
> [root@centos-01 ~]# cat employee.json
> {
>   "firstName": "John",
>   "lastName": "Smith",
>   "isAlive": true,
>   "age": 45,
>   "height_cm": 177.6,
>   "address": {
> "streetAddress": "29 4th Street",
> "city": "New York",
> "state": "NY",
> "postalCode": "10021-3100"
>   },
>   "phoneNumbers": [
> {
>   "type": "home",
>   "number": "212 555-1234"
> },
> {
>   "type": "office",
>   "number": "646 555-4567"
> }
>   ],
>   "children": [],
>   "hobbies": ["scuba diving","hiking","biking","rock climbing","surfing"]
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-3562:
-
Labels:   (was: ready-to-commit)

> Query fails when using flatten on JSON data where some documents have an 
> empty array
> 
>
> Key: DRILL-3562
> URL: https://issues.apache.org/jira/browse/DRILL-3562
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.1.0
>Reporter: Philip Deegan
>Assignee: Serhii Harnyk
> Fix For: Future
>
>
> Drill query fails when using flatten when some records contain an empty array 
> {noformat}
> SELECT COUNT(*) FROM (SELECT FLATTEN(t.a.b.c) AS c FROM dfs.`flat.json` t) 
> flat WHERE flat.c.d.e = 'f' limit 1;
> {noformat}
> Succeeds on 
> { "a": { "b": { "c": [  { "d": {  "e": "f" } } ] } } }
> Fails on
> { "a": { "b": { "c": [] } } }
> Error
> {noformat}
> Error: SYSTEM ERROR: ClassCastException: Cannot cast 
> org.apache.drill.exec.vector.NullableIntVector to 
> org.apache.drill.exec.vector.complex.RepeatedValueVector
> {noformat}
> Is it possible to ignore the empty arrays, or do they need to be populated 
> with dummy data?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-5164) equi-join query results in CompileException

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-5164:


Assignee: Serhii Harnyk

> equi-join query results in CompileException
> ---
>
> Key: DRILL-5164
> URL: https://issues.apache.org/jira/browse/DRILL-5164
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Attachments: manyColsInJson.json
>
>
> Drill 1.9.0 
> git commit ID : 4c1b420b
> 4 node CentOS cluster
> JSON file has 4095 keys (columns)
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from `manyColsInJson.json` t1, 
> `manyColsInJson.json` t2 where t1.key2000 = t2.key2000;
> Error: SYSTEM ERROR: CompileException: File 
> 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
>  Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
> public void doSetup(FragmentContext context, VectorContainer buildBatch, 
> RecordBatch probeBatch, RecordBatch outgoing)
> ^ (compiler.err.limit.code)
> Fragment 0:0
> [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-12-26 09:52:11,321 [279f17fd-c8f0-5d18-1124-76099f0a5cc8:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: CompileException: File 
> 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
>  Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
> public void doSetup(FragmentContext context, VectorContainer buildBatch, 
> RecordBatch probeBatch, RecordBatch outgoing)
> ^ (compiler.err.limit.code)
> Fragment 0:0
> [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> CompileException: File 
> 'org.apache.drill.exec.compile.DrillJavaFileObject[HashJoinProbeGen294.java]',
>  Line 16397, Column 17: HashJoinProbeGen294.java:16397: error: code too large
> public void doSetup(FragmentContext context, VectorContainer buildBatch, 
> RecordBatch probeBatch, RecordBatch outgoing)
> ^ (compiler.err.limit.code)
> Fragment 0:0
> [Error Id: 7d0efa7e-e183-4c40-939a-4908699f94bf on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
>  [drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.9.0.jar:1.9.0]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: 
> org.apache.drill.exec.exception.SchemaChangeException: 
> org.apache.drill.exec.exception.ClassTransformationException: 
> java.util.concurrent.ExecutionException: 
> org.apache.drill.exec.exception.ClassTransformationException: Failure 
> generating transformation classes for value:
> package org.apache.drill.exec.test.generated;
> ...
> public class HashJoinProbeGen294 {
> NullableVarCharVector[] vv0;
> NullableVarCharVector vv3;
> NullableVarCharVector[] vv6;
> ...
> vv49137 .copyFromSafe((probeIndex), (outIndex), vv49134);
> vv49143 .copyFromSafe((probeIndex), (outIndex), vv49140);
> vv49149 .copyFromSafe((probeIndex), (outIndex), vv49146);
> }
> }
> 
> public void __DRILL_INIT__()
> throws SchemaChangeException
> {
> }
> }
> at 
> org.apache.drill.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:302)
>  ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:78) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> org.apache.drill.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:74) 
> ~[drill-java-exec-1.9.0.jar:1.9.0]
> at 
> 

[jira] [Commented] (DRILL-3562) Query fails when using flatten on JSON data where some documents have an empty array

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763911#comment-15763911
 ] 

Serhii Harnyk commented on DRILL-3562:
--

This issue is still reproducing on the Json file:
{noformat}
{ "a": { "b": { "c": [] } } }
{noformat}
with query:
{code}
select FLATTEN(t.a.b.c) AS c from dfs.`/home/files/flat.json` t;
{code}

> Query fails when using flatten on JSON data where some documents have an 
> empty array
> 
>
> Key: DRILL-3562
> URL: https://issues.apache.org/jira/browse/DRILL-3562
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.1.0
>Reporter: Philip Deegan
>Assignee: Serhii Harnyk
> Fix For: Future
>
>
> Drill query fails when using flatten when some records contain an empty array 
> {noformat}
> SELECT COUNT(*) FROM (SELECT FLATTEN(t.a.b.c) AS c FROM dfs.`flat.json` t) 
> flat WHERE flat.c.d.e = 'f' limit 1;
> {noformat}
> Succeeds on 
> { "a": { "b": { "c": [  { "d": {  "e": "f" } } ] } } }
> Fails on
> { "a": { "b": { "c": [] } } }
> Error
> {noformat}
> Error: SYSTEM ERROR: ClassCastException: Cannot cast 
> org.apache.drill.exec.vector.NullableIntVector to 
> org.apache.drill.exec.vector.complex.RepeatedValueVector
> {noformat}
> Is it possible to ignore the empty arrays, or do they need to be populated 
> with dummy data?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5032:
-
Labels:   (was: ready-to-commit)

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
> ~[protobuf-java-2.5.0.jar:na]
> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
> 

[jira] [Resolved] (DRILL-1808) Large compilation unit tests fails due to high memory allocation

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk resolved DRILL-1808.
--
   Resolution: Duplicate
Fix Version/s: (was: Future)
   1.10.0

Fixed in 
[810198b|https://github.com/apache/drill/commit/810198b18bfbe38712256c58b102bca079d934c1]

> Large compilation unit tests fails due to high memory allocation
> 
>
> Key: DRILL-1808
> URL: https://issues.apache.org/jira/browse/DRILL-1808
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Reporter: Steven Phillips
>Priority: Minor
> Fix For: 1.10.0
>
>
> The buildSchema method of external sort calls allocateNew() on each vector in 
> schema batch, which normally isn't much memory, but when there are thousands 
> of columns, this ends up being significant. We should avoid allocating any 
> memory for the schema batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5117) Compile error when query a json file with 1000+columns

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763699#comment-15763699
 ] 

Serhii Harnyk commented on DRILL-5117:
--

Fixed in 
[810198b|https://github.com/apache/drill/commit/810198b18bfbe38712256c58b102bca079d934c1]

> Compile error when query a json file with 1000+columns
> --
>
> Key: DRILL-5117
> URL: https://issues.apache.org/jira/browse/DRILL-5117
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
>
> Query failed with compile error when we querying a json file with 
> 1000+columns:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
> Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class 
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from sqlline.log:
> {noformat}
> 2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 27b54af4-b41f-0682-e50d-626de4eff68e: select * from 
> dfs.`/tmp/tooManyFields.json` limit 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> localhost.  Skipping affinity to that host.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, 
> Average start: 6.956000 μs .
> 2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State to report: RUNNING
> 2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN  
> o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - 
> probably you're running Drill with a JRE and not a JDK
> 2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested RUNNING --> 
> FAILED
> 2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested FAILED --> 
> FINISHED
> 2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: 
> Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" 
> grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> JaninoRuntimeException: Code attribute in class 
> "org.apache.drill.exec.test.generated.CopierGen0" 

[jira] [Updated] (DRILL-5117) Compile error when query a json file with 1000+columns

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5117:
-
Labels:   (was: ready-to-commit)

> Compile error when query a json file with 1000+columns
> --
>
> Key: DRILL-5117
> URL: https://issues.apache.org/jira/browse/DRILL-5117
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
>
> Query failed with compile error when we querying a json file with 
> 1000+columns:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
> Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class 
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from sqlline.log:
> {noformat}
> 2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 27b54af4-b41f-0682-e50d-626de4eff68e: select * from 
> dfs.`/tmp/tooManyFields.json` limit 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
> localhost.  Skipping affinity to that host.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
> 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, 
> Average start: 6.956000 μs .
> 2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State to report: RUNNING
> 2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN  
> o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - 
> probably you're running Drill with a JRE and not a JDK
> 2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested RUNNING --> 
> FAILED
> 2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested FAILED --> 
> FINISHED
> 2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: 
> Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" 
> grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> JaninoRuntimeException: Code attribute in class 
> "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB
> Fragment 0:0
> [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on 

[jira] [Updated] (DRILL-4864) Add ANSI format for date/time functions

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4864:
-
Description: 
The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
layer. This is not following SQL conventions used by ANSI and many other 
database engines on the market.

Add new UDFs: 

* sql_to_date(String, Format), 
* sql_to_time(String, Format), 
* sql_to_timestamp(String, Format)

that requires Postgres datetime format.


Table of supported Postgres patterns
||Pattern name||Postgres format   
|Full name of day|day   
|Day of year|ddd   
|Day of month|dd
|Day of week|d   
|Name of month|month
|Abr name of month|mon
|Full era name|ee
|Name of day|dy   
|Time zone|tz   
|Hour 12 |hh   
|Hour 12 |hh12   
|Hour 24|hh24
|Minute of hour|mi  
|Second of minute|ss   
|Millisecond of minute|ms
|Week of year|ww   
|Month|mm   
|Halfday am|am
|Year   |   y   
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html   |


Table of acceptable Postgres pattern modifiers, which may be used in Format 
string
||Description||Pattern||
|fill mode (suppress padding blanks and zeroes)|fm |
|fixed format global option (see usage notes)|fx |
|translation mode (print localized day and month names based on 
lc_messages)|tm |
|spell mode (not yet implemented)|sp|
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html|

  was:
The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
layer. This is not following SQL conventions used by ANSI and many other 
database engines on the market.

Add new UDFs: 

* sql_to_date(String, Format), 
* sql_to_time(String, Format), 
* sql_to_timestamp(String, Format)

that requires Postgres datetime format.


Table of supported Postgres patterns
||Pattern name||Postgres format   
|Full name of day|day   
|Day of year|ddd   
|Day of month|dd
|Day of week|d   
|Name of month|month
|Abr name of month|mon
|Full era name|ee
|Name of day|dy   
|Time zone|tz   
|Hour 12 |hh   
|Hour 12 |hh12   
|Hour 24|hh24
|Minute of hour|mi  
|Second of minute|ss   
|Millisecond of minute|ms
|Week of year|ww   
|Month|mm   
|Halfday am|am
|Year   |   y   
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html   




Table of acceptable Postgres pattern modifiers, which may be used in Format 
string
||Description||Pattern||
|fill mode (suppress padding blanks and zeroes)|fm |
|fixed format global option (see usage notes)|fx |
|translation mode (print localized day and month names based on 
lc_messages)|tm |
|spell mode (not yet implemented)|sp|
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html|


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by ANSI and many other 
> database engines on the market.
> Add new UDFs: 
> * sql_to_date(String, Format), 
> * sql_to_time(String, Format), 
> * sql_to_timestamp(String, Format)
> that requires Postgres datetime format.
> Table of supported Postgres patterns
> ||Pattern name||Postgres format   
> |Full name of day|day   
> |Day of year|ddd   
> |Day of month|dd
> |Day of week|d   
> |Name of month|month
> |Abr name of month|mon
> |Full era name|ee
> |Name of day|dy   
> |Time zone|tz   
> |Hour 12 |hh   
> |Hour 12 |hh12   
> |Hour 24|hh24
> |Minute of hour|mi  
> |Second of minute|ss   
> |Millisecond of minute|ms
> |Week of year|ww   
> |Month|mm   
> |Halfday am|  

[jira] [Updated] (DRILL-4864) Add ANSI format for date/time functions

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4864:
-
Description: 
The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
layer. This is not following SQL conventions used by ANSI and many other 
database engines on the market.

Add new UDFs: 

* sql_to_date(String, Format), 
* sql_to_time(String, Format), 
* sql_to_timestamp(String, Format)

that requires Postgres datetime format.


Table of supported Postgres patterns
||Pattern name||Postgres format   
|Full name of day|day   
|Day of year|ddd   
|Day of month|dd
|Day of week|d   
|Name of month|month
|Abr name of month|mon
|Full era name|ee
|Name of day|dy   
|Time zone|tz   
|Hour 12 |hh   
|Hour 12 |hh12   
|Hour 24|hh24
|Minute of hour|mi  
|Second of minute|ss   
|Millisecond of minute|ms
|Week of year|ww   
|Month|mm   
|Halfday am|am
|Year   |   y   
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html   




Table of acceptable Postgres pattern modifiers, which may be used in Format 
string
||Description||Pattern||
|fill mode (suppress padding blanks and zeroes)|fm |
|fixed format global option (see usage notes)|fx |
|translation mode (print localized day and month names based on 
lc_messages)|tm |
|spell mode (not yet implemented)|sp|
|ref.|
https://www.postgresql.org/docs/8.2/static/functions-formatting.html|

  was:
The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
layer. This is not following SQL conventions used by ANSI and many other 
database engines on the market.

Add new UDF "ansi_to_joda(string)", that takes string that represents ANSI 
datetime format and returns string that represents equal Joda format.
Add new session option "drill.exec.fn.to_date_format" that can be one of two 
values - "JODA"(default) and "ANSI".
If option is set to "JODA" queries with to_date() function would work in usual 
way.
If option is set to "ANSI" second argument would be wrapped with ansi_to_joda() 
function, that allows user to use ANSI datetime format
Wrapping is used in to_date(), to_time() and to_timestamp() functions.
Table of joda and ansi patterns which may be replaced
||  Pattern name||  Ansi format ||  JodaTime format
|   Full name of day|   day |   
|   Day of year |   ddd |   D
|   Day of month|   dd  |   d
|   Day of week |   d   |   e
|   Name of month   |   month   |   
|   Abr name of month   |   mon |   MMM
|   Full era name   |   ee  |   G
|   Name of day |   dy  |   E
|   Time zone   |   tz  |   TZ
|   Hour 12 |   hh  |   h
|   Hour 12 |   hh12|   h
|   Hour 24 |   hh24|   H
|   Minute of hour  |   mi  |   m
|   Second of minute|   ss  |   s
|   Millisecond of minute   |   ms  |   S
|   Week of year|   ww  |   w
|   Month   |   mm  |   MM
|   Halfday am  |   am  |   aa
|   Halfday pm  |   pm  |   aa
|   ref.|   
https://www.postgresql.org/docs/8.2/static/functions-formatting.html|   
http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html |


Table of ansi pattern modifiers, which may be deleted from string
||  Description ||  Pattern ||
|   fill mode (suppress padding blanks and zeroes)  |   fm  |
|   fixed format global option (see usage notes)|   fx  |
|   translation mode (print localized day and month names based on 
lc_messages) |   tm  |
|   spell mode (not yet implemented)|   sp  |
|   ref.|   
https://www.postgresql.org/docs/8.2/static/functions-formatting.html|


> Add ANSI format for date/time functions
> ---
>
> Key: DRILL-4864
> URL: https://issues.apache.org/jira/browse/DRILL-4864
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: doc-impacting
> Fix For: Future
>
>
> The TO_DATE() is exposing the Joda string formatting conventions into the SQL 
> layer. This is not following SQL conventions used by 

[jira] [Updated] (DRILL-5048) Fix type mismatch error in case statement with null timestamp

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5048:
-
Summary: Fix type mismatch error in case statement with null timestamp  
(was: AssertionError when case statement is used with timestamp and null)

> Fix type mismatch error in case statement with null timestamp
> -
>
> Key: DRILL-5048
> URL: https://issues.apache.org/jira/browse/DRILL-5048
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
>
> AssertionError when we use case with timestamp and null:
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
> . . . . . . . . . . . . . . > FROM
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
> . . . . . . . . . . . . . . > FROM (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> [Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> ... 16 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4938) Report UserException when constant expression reduction fails

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4938:
-
Summary: Report UserException when constant expression reduction fails  
(was: need better error message)

> Report UserException when constant expression reduction fails
> -
>
> Key: DRILL-4938
> URL: https://issues.apache.org/jira/browse/DRILL-4938
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Minor
> Fix For: 1.10.0
>
>
> We need a better error message instead of DrillRuntimeException
> Drill 1.9.0 git commit ID : 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 2016/09/22) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing 
> expression in constant expression evaluator [CASE(false, =(null, /(/(2016, 
> 9), 22)), =(CAST('2016/09/22'):DATE NOT NULL, /(/(2016, 9), 22)))].  Errors:
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4938) need better error message

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4938:
-
Labels:   (was: ready-to-commit)

> need better error message
> -
>
> Key: DRILL-4938
> URL: https://issues.apache.org/jira/browse/DRILL-4938
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Minor
> Fix For: 1.10.0
>
>
> We need a better error message instead of DrillRuntimeException
> Drill 1.9.0 git commit ID : 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 2016/09/22) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing 
> expression in constant expression evaluator [CASE(false, =(null, /(/(2016, 
> 9), 22)), =(CAST('2016/09/22'):DATE NOT NULL, /(/(2016, 9), 22)))].  Errors:
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-5048) AssertionError when case statement is used with timestamp and null

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5048:
-
Labels:   (was: ready-to-commit)

> AssertionError when case statement is used with timestamp and null
> --
>
> Key: DRILL-5048
> URL: https://issues.apache.org/jira/browse/DRILL-5048
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: 1.10.0
>
>
> AssertionError when we use case with timestamp and null:
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
> . . . . . . . . . . . . . . > FROM
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
> . . . . . . . . . . . . . . > FROM (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> [Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> ... 16 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5117) Compile error when query a json file with 1000+columns

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5117:


 Summary: Compile error when query a json file with 1000+columns
 Key: DRILL-5117
 URL: https://issues.apache.org/jira/browse/DRILL-5117
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Codegen
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


Query failed with compile error when we querying a json file with 1000+columns:
{noformat}
0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1;
Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] 
(state=,code=0)
{noformat}

Stack trace from sqlline.log:
{noformat}
2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
27b54af4-b41f-0682-e50d-626de4eff68e: select * from 
dfs.`/tmp/tooManyFields.json` limit 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host 
localhost.  Skipping affinity to that host.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 13ms total, 13.922965ms avg, 13ms max.
2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, Average 
start: 6.956000 μs .
2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State to report: RUNNING
2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN  
o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - 
probably you're running Drill with a JRE and not a JDK
2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested RUNNING --> FAILED
2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: 
State change requested FAILED --> FINISHED
2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: 
Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows 
beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
JaninoRuntimeException: Code attribute in class 
"org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB

Fragment 0:0

[Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
 ~[drill-common-1.8.0.jar:1.8.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
 [drill-java-exec-1.8.0.jar:1.8.0]
at 

[jira] [Closed] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk closed DRILL-4941.

Resolution: Won't Fix

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-4941:
-
Fix Version/s: (was: 1.9.0)

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728785#comment-15728785
 ] 

Serhii Harnyk commented on DRILL-4941:
--

So all this queries should fail. To execute all this queries you should cast 
nulls to appropriate types. For example queries:
{noformat}
SELECT (CASE WHEN true or cast(null as boolean) then 1 else 0 end) from 
(VALUES(1))
{noformat}
{noformat}
select NULLIF(-1,cast(null as integer))  from (values(1)) as foo
{noformat}
works and returns right result.

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728760#comment-15728760
 ] 

Serhii Harnyk commented on DRILL-4941:
--

Julian Hyde mentioned:
The SQL standard allows NULL values if you can determine the type, e.g.

{code}
  INSERT INTO Emp (empno, commission) VALUES (100, NULL)
{code}

but not naked NULLs:

{code}
  SELECT empno, NULL FROM Emp
{code}

You have to cast them:

{code}
  SELECT empno, CAST(NULL AS INTEGER) From Emp
{code}

You may think that we should allow NULL in places where we can deduce the 
intended type because there is only one overloaded operator that matches. This 
would be the case with say

{code}
  SELECT * FROM Emp
  WHERE empno > 100 OR NULL
{code}

But the SQL standard doesn't allow it. Postgres goes beyond the standard. 
Calcite goes beyond the standard in some cases, and I'd consider going beyond 
it here. But to make a special case just for BOOLEAN doesn't make sense.

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 

[jira] [Assigned] (DRILL-4941) UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4941:


Assignee: Serhii Harnyk

> UnsupportedOperationException : CASE WHEN true or null then 1 else 0 end
> 
>
> Key: DRILL-4941
> URL: https://issues.apache.org/jira/browse/DRILL-4941
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
> Fix For: 1.9.0
>
>
> Below case expression results in UnsupportedOperationException on Drill 1.9.0 
> git commit ID: 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or null then 1 else 0 
> end) from (VALUES(1));
> Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlLiteral: NULL
> SQL Query null
> [Error Id: 822ec7b0-3630-478c-b82a-0acedc39a560 on centos-01.qa.lab:31010] 
> (state=,code=0)
> -- changing null to "not null" in the search condition causes Drill to return 
> results
> 0: jdbc:drill:schema=dfs.tmp> SELECT (CASE WHEN true or not null then 1 else 
> 0 end) from (VALUES(1));
> +-+
> | EXPR$0  |
> +-+
> | 1   |
> +-+
> 1 row selected (0.11 seconds)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.UnsupportedOperationException: class 
> org.apache.calcite.sql.SqlLiteral: NULL
> at org.apache.calcite.util.Util.needToImplement(Util.java:920) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.getValidatedNodeType(SqlValidatorImpl.java:1426)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.adjustType(SqlBinaryOperator.java:103)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.deriveType(SqlOperator.java:511) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlBinaryOperator.deriveType(SqlBinaryOperator.java:143)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.checkOperandTypes(SqlCaseOperator.java:178)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:430) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.fun.SqlCaseOperator.deriveType(SqlCaseOperator.java:164)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4337)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4324)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1501)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1484)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:446)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4942) incorrect result - case when (not null is null) then true else false end

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712377#comment-15712377
 ] 

Serhii Harnyk commented on DRILL-4942:
--

The problem is in Calcite - CALCITE-1095.
It was fixed in Calcite 1.9.0.
When Drill migrates to Calcite 1.9.0 or higher, issue will be resolved.

> incorrect result  - case when (not null is null) then true else false end
> -
>
> Key: DRILL-4942
> URL: https://issues.apache.org/jira/browse/DRILL-4942
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Drill's result is different from that of Calcite and Postgres result.
> Postgres 9.3 results
> {noformat}
> postgres=# select (case when (not null is null) then true else false end) 
> from (values(1)) foo;
>  case
> --
>  f
> (1 row)
> {noformat}
> Calcite release 1.8.0 too returns false
> {noformat}
> 0: jdbc:calcite:model=target/test-classes/mod> select (case when (not null is 
> null) then true else false end) from (values(1));
> ++
> | EXPR$0 |
> ++
> | false  |
> ++
> 1 row selected (1.271 seconds)
> {noformat}
> Drill 1.9.0 git commit ID: 4edabe7a, returns true
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when (not null is null) then true 
> else false end) from (values(1));
> +-+
> | EXPR$0  |
> +-+
> | true|
> +-+
> 1 row selected (0.096 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4942) incorrect result - case when (not null is null) then true else false end

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4942:


Assignee: Serhii Harnyk

> incorrect result  - case when (not null is null) then true else false end
> -
>
> Key: DRILL-4942
> URL: https://issues.apache.org/jira/browse/DRILL-4942
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Drill's result is different from that of Calcite and Postgres result.
> Postgres 9.3 results
> {noformat}
> postgres=# select (case when (not null is null) then true else false end) 
> from (values(1)) foo;
>  case
> --
>  f
> (1 row)
> {noformat}
> Calcite release 1.8.0 too returns false
> {noformat}
> 0: jdbc:calcite:model=target/test-classes/mod> select (case when (not null is 
> null) then true else false end) from (values(1));
> ++
> | EXPR$0 |
> ++
> | false  |
> ++
> 1 row selected (1.271 seconds)
> {noformat}
> Drill 1.9.0 git commit ID: 4edabe7a, returns true
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when (not null is null) then true 
> else false end) from (values(1));
> +-+
> | EXPR$0  |
> +-+
> | true|
> +-+
> 1 row selected (0.096 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690731#comment-15690731
 ] 

Serhii Harnyk edited comment on DRILL-5032 at 11/23/16 5:24 PM:


[~cshi] plz, can you add "+1" in comment in PR?


was (Author: sharnyk):
[~cshi] plz, can you add "+1" in commit in PR?

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>  Labels: ready-to-commit
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> 

[jira] [Updated] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5032:
-
Labels: К  (was: )

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Chunhui Shi
>  Labels: К
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
> ~[protobuf-java-2.5.0.jar:na]
> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> 

[jira] [Commented] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690731#comment-15690731
 ] 

Serhii Harnyk commented on DRILL-5032:
--

[~cshi] plz, can you add "+1" in commit in PR?

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Chunhui Shi
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
> ~[protobuf-java-2.5.0.jar:na]
> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
> 

[jira] [Commented] (DRILL-4764) Parquet file with INT_16, etc. logical types not supported by simple SELECT

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690349#comment-15690349
 ] 

Serhii Harnyk commented on DRILL-4764:
--

[~paul-rogers], could you tell me how did you generate this parquet files?

> Parquet file with INT_16, etc. logical types not supported by simple SELECT
> ---
>
> Key: DRILL-4764
> URL: https://issues.apache.org/jira/browse/DRILL-4764
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Serhii Harnyk
> Attachments: int_16.parquet, int_8.parquet, uint_16.parquet, 
> uint_32.parquet, uint_8.parquet
>
>
> Create a Parquet file with the following schema:
> message int16Data { required int32 index; required int32 value (INT_16); }
> Store it as int_16.parquet in the local file system. Query it with:
> SELECT * from `local`.`root`.`int_16.parquet`;
> The result, in the web UI, is this error:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> UnsupportedOperationException: unsupported type: INT32 INT_16 Fragment 0:0 
> [Error Id: c63f66b4-e5a9-4a35-9ceb-546b74645dd4 on 172.30.1.28:31010]
> The INT_16 logical (or "original") type simply tells consumers of the file 
> that the data is actually a 16-bit signed int. Presumably, this should tell 
> Drill to use the SmallIntVector (or NullableSmallIntVector) class for 
> storage. Without supporting this annotation, even 16-bit integers must be 
> stored as 32-bits within Drill.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4764) Parquet file with INT_16, etc. logical types not supported by simple SELECT

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4764:


Assignee: Serhii Harnyk

> Parquet file with INT_16, etc. logical types not supported by simple SELECT
> ---
>
> Key: DRILL-4764
> URL: https://issues.apache.org/jira/browse/DRILL-4764
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Serhii Harnyk
> Attachments: int_16.parquet, int_8.parquet, uint_16.parquet, 
> uint_32.parquet, uint_8.parquet
>
>
> Create a Parquet file with the following schema:
> message int16Data { required int32 index; required int32 value (INT_16); }
> Store it as int_16.parquet in the local file system. Query it with:
> SELECT * from `local`.`root`.`int_16.parquet`;
> The result, in the web UI, is this error:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> UnsupportedOperationException: unsupported type: INT32 INT_16 Fragment 0:0 
> [Error Id: c63f66b4-e5a9-4a35-9ceb-546b74645dd4 on 172.30.1.28:31010]
> The INT_16 logical (or "original") type simply tells consumers of the file 
> that the data is actually a 16-bit signed int. Presumably, this should tell 
> Drill to use the SmallIntVector (or NullableSmallIntVector) class for 
> storage. Without supporting this annotation, even 16-bit integers must be 
> stored as 32-bits within Drill.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4217) Query parquet file treat INT_16 & INT_8 as INT32

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4217:


Assignee: Serhii Harnyk

> Query parquet file treat INT_16 & INT_8 as INT32
> 
>
> Key: DRILL-4217
> URL: https://issues.apache.org/jira/browse/DRILL-4217
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Data Types
>Reporter: Low Chin Wei
>Assignee: Serhii Harnyk
>
> Encounter this issue while trying to query a parquet file:
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> UnsupportedOperationException: unsupported type: INT32 INT_16 Fragment 1:1 
> We can treat the following Field Type as INTEGER before support of Short & 
> Byte is implemeted: 
> - INT32 INT_16
> - INT32 INT_8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4923) Use of CASE WHEN inside a sub-query results in AssertionError

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690303#comment-15690303
 ] 

Serhii Harnyk commented on DRILL-4923:
--

The problem is in Calcite - CALCITE-1502.
It will be fixed in Calcite 1.11.0.
When Drill migrates to Calcite 1.11.0 or higher, issue will be resolved.

> Use of CASE WHEN inside a sub-query results in AssertionError
> -
>
> Key: DRILL-4923
> URL: https://issues.apache.org/jira/browse/DRILL-4923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.9.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Attachments: 0_0_0.parquet
>
>
> Use of CASE WHEN inside a sub-query results in AssertionError
> Drill 1.9.0 git commit ID: f3c26e34
> parquet file used in test is attached
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from (SELECT CASE WHEN 'WA'='WA' THEN 
> '13' ELSE CAST(state as varchar(2)) end as state_nm from `emp_tbl` as a) 
> `LOG_FCT` where ('WA'=`LOG_FCT`.`state_nm`);
> Error: SYSTEM ERROR: AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-10-03 09:37:32,756 [280dd922-b97d-2ccd-551c-e425c075d91f:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type 
> mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935) 
> [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) 
> [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ProjectMergeRule:force_mode, args 
> [rel#5:LogicalProject.NONE.ANY([]).[](input=rel#11103:Subset#6.NONE.ANY([]).[],state_nm='13'),
>  
> rel#11107:LogicalProject.NONE.ANY([]).[](input=rel#11095:Subset#4.NONE.ANY([]).[],state=$1)]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ProjectMergeRule:force_mode, args 
> [rel#5:LogicalProject.NONE.ANY([]).[](input=rel#11103:Subset#6.NONE.ANY([]).[],state_nm='13'),
>  
> rel#11107:LogicalProject.NONE.ANY([]).[](input=rel#11095:Subset#4.NONE.ANY([]).[],state=$1)]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> 

[jira] [Updated] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk updated DRILL-5032:
-
Attachment: plan with fix
plan

Physical plan before and after fix

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Chunhui Shi
> Attachments: plan, plan with fix
>
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
> ~[protobuf-java-2.5.0.jar:na]
> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
> 

[jira] [Commented] (DRILL-4842) SELECT * on JSON data results in NumberFormatException

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684086#comment-15684086
 ] 

Serhii Harnyk commented on DRILL-4842:
--

[~cshi] there is no bug for 1000 nulls. Our fix is aimed for handling columns 
from bath to batch. 
The problem is in JsonReader. I tested it in parquet, there is no such bug, due 
to parquet handling columns metadata.
The root of problem is that before fix JsonReader didn't handle nullable 
columns.

> SELECT * on JSON data results in NumberFormatException
> --
>
> Key: DRILL-4842
> URL: https://issues.apache.org/jira/browse/DRILL-4842
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Khurram Faraaz
>Assignee: Chunhui Shi
> Attachments: tooManyNulls.json
>
>
> Note that doing SELECT c1 returns correct results, the failure is seen when 
> we do SELECT star. json.all_text_mode was set to true.
> JSON file tooManyNulls.json has one key c1 with 4096 nulls as its value and 
> the 4097th key c1 has the value "Hello World"
> git commit ID : aaf220ff
> MapR Drill 1.8.0 RPM
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> alter session set 
> `store.json.all_text_mode`=true;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | store.json.all_text_mode updated.  |
> +---++
> 1 row selected (0.27 seconds)
> 0: jdbc:drill:schema=dfs.tmp> SELECT c1 FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> +--+
> |  c1  |
> +--+
> | Hello World  |
> +--+
> 1 row selected (0.243 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select * FROM `tooManyNulls.json` WHERE c1 IN 
> ('Hello World');
> Error: SYSTEM ERROR: NumberFormatException: Hello World
> Fragment 0:0
> [Error Id: 9cafb3f9-3d5c-478a-b55c-900602b8765e on centos-01.qa.lab:31010]
>  (java.lang.NumberFormatException) Hello World
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.nfeI():95
> 
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.varTypesToInt():120
> org.apache.drill.exec.test.generated.FiltererGen1169.doSetup():45
> org.apache.drill.exec.test.generated.FiltererGen1169.setup():54
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer():195
> 
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema():107
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():78
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():135
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():104
> 
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
> org.apache.drill.exec.physical.impl.BaseRootExec.next():94
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745 (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.NumberFormatException: Hello World
> at 
> 

[jira] [Commented] (DRILL-4936) case expression when true then null, fails

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683630#comment-15683630
 ] 

Serhii Harnyk commented on DRILL-4936:
--

[~khfaraaz] 

There is a comment in Calcite code:
// according to the sql standard we can not have all of the THEN
// statements and the ELSE returning null

I checked spec and looks they are right: ISO/IEC 9075-2:2011(E) 6.12  - Syntax Rules:
3) At least one  in a  shall specify a  


> case expression when true then null, fails
> --
>
> Key: DRILL-4936
> URL: https://issues.apache.org/jira/browse/DRILL-4936
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Postgres compile and executes same query and returns null, whereas Drill 
> returns error.
> Drill 1.9.0
> git commit id : 4edabe7a
> Drom Drill 1.9.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when true then null else null end) 
> from (values(1));
> Error: VALIDATION ERROR: From line 1, column 9 to line 1, column 46: ELSE 
> clause or at least one THEN clause must be non-NULL
> SQL Query null
> [Error Id: 2912d0b5-b947-4cd7-b325-83c809e5ae50 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when true then null end) from 
> (values(1));
> Error: VALIDATION ERROR: From line 1, column 9 to line 1, column 36: ELSE 
> clause or at least one THEN clause must be non-NULL
> SQL Query null
> [Error Id: cdb56929-b175-4474-b8a4-e6eec4d4a28f on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> from Postgres 9.3
> {noformat}
> postgres=# select (case when true then null else null end) from (values(1)) 
> foo;
>  case
> --
> (1 row)
> {noformat}
> {noformat}
> postgres=# select (case when true then null end) from (values(1)) foo;
>  case
> --
> (1 row)
> {noformat}  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4938) need better error message

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677020#comment-15677020
 ] 

Serhii Harnyk commented on DRILL-4938:
--

[~khfaraaz]
DrillRuntimeException throws in DrillConstExecutor and is common exception for 
a lot of runtime exceptions that are collected by ErrorCollector.
We can create another Exception, but it should have enough abstract name.

> need better error message
> -
>
> Key: DRILL-4938
> URL: https://issues.apache.org/jira/browse/DRILL-4938
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Minor
>
> We need a better error message instead of DrillRuntimeException
> Drill 1.9.0 git commit ID : 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 2016/09/22) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing 
> expression in constant expression evaluator [CASE(false, =(null, /(/(2016, 
> 9), 22)), =(CAST('2016/09/22'):DATE NOT NULL, /(/(2016, 9), 22)))].  Errors:
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4938) need better error message

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4938:


Assignee: Serhii Harnyk

> need better error message
> -
>
> Key: DRILL-4938
> URL: https://issues.apache.org/jira/browse/DRILL-4938
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Minor
>
> We need a better error message instead of DrillRuntimeException
> Drill 1.9.0 git commit ID : 4edabe7a
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (res1 = 2016/09/22) res2
> . . . . . . . . . . . . . . > from
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > select (case when (false) then null else 
> cast('2016/09/22' as date) end) res1
> . . . . . . . . . . . . . . > from (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing 
> expression in constant expression evaluator [CASE(false, =(null, /(/(2016, 
> 9), 22)), =(CAST('2016/09/22'):DATE NOT NULL, /(/(2016, 9), 22)))].  Errors:
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> Error in expression at index -1.  Error: Missing function implementation: 
> [castTIMESTAMP(INT-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4936) case expression when true then null, fails

[Serhii Harnyk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Serhii Harnyk reassigned DRILL-4936:


Assignee: Serhii Harnyk

> case expression when true then null, fails
> --
>
> Key: DRILL-4936
> URL: https://issues.apache.org/jira/browse/DRILL-4936
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>
> Postgres compile and executes same query and returns null, whereas Drill 
> returns error.
> Drill 1.9.0
> git commit id : 4edabe7a
> Drom Drill 1.9.0
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when true then null else null end) 
> from (values(1));
> Error: VALIDATION ERROR: From line 1, column 9 to line 1, column 46: ELSE 
> clause or at least one THEN clause must be non-NULL
> SQL Query null
> [Error Id: 2912d0b5-b947-4cd7-b325-83c809e5ae50 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select (case when true then null end) from 
> (values(1));
> Error: VALIDATION ERROR: From line 1, column 9 to line 1, column 36: ELSE 
> clause or at least one THEN clause must be non-NULL
> SQL Query null
> [Error Id: cdb56929-b175-4474-b8a4-e6eec4d4a28f on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> from Postgres 9.3
> {noformat}
> postgres=# select (case when true then null else null end) from (values(1)) 
> foo;
>  case
> --
> (1 row)
> {noformat}
> {noformat}
> postgres=# select (case when true then null end) from (values(1)) foo;
>  case
> --
> (1 row)
> {noformat}  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5048) AssertionError when case statement is used with timestamp and null

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673472#comment-15673472
 ] 

Serhii Harnyk commented on DRILL-5048:
--

Originally found by [~khfaraaz] in DRILL-4923

> AssertionError when case statement is used with timestamp and null
> --
>
> Key: DRILL-5048
> URL: https://issues.apache.org/jira/browse/DRILL-5048
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
> Fix For: Future
>
>
> AssertionError when we use case with timestamp and null:
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
> . . . . . . . . . . . . . . > FROM
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
> CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
> . . . . . . . . . . . . . . > FROM (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar;
> Error: SYSTEM ERROR: AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> [Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
> rowtype of set:
> RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
> ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
>  ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
> ... 16 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4923) Use of CASE WHEN inside a sub-query results in AssertionError

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673457#comment-15673457
 ] 

Serhii Harnyk commented on DRILL-4923:
--

[~khfaraaz]
The problem with case statement when using timestamp and null is not related to 
this Jira, so I have created separate Jira to track this issue: DRILL-5048

> Use of CASE WHEN inside a sub-query results in AssertionError
> -
>
> Key: DRILL-4923
> URL: https://issues.apache.org/jira/browse/DRILL-4923
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.9.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Attachments: 0_0_0.parquet
>
>
> Use of CASE WHEN inside a sub-query results in AssertionError
> Drill 1.9.0 git commit ID: f3c26e34
> parquet file used in test is attached
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select * from (SELECT CASE WHEN 'WA'='WA' THEN 
> '13' ELSE CAST(state as varchar(2)) end as state_nm from `emp_tbl` as a) 
> `LOG_FCT` where ('WA'=`LOG_FCT`.`state_nm`);
> Error: SYSTEM ERROR: AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-10-03 09:37:32,756 [280dd922-b97d-2ccd-551c-e425c075d91f:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Type 
> mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(CHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" NOT NULL state_nm) NOT NULL
> rowtype of set:
> RecordType(VARCHAR(2) CHARACTER SET "ISO-8859-1" COLLATE 
> "ISO-8859-1$en_US$primary" state_nm) NOT NULL
> [Error Id: 59b8da55-cf01-41fe-ba7a-018f7fad2892 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:825)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:935) 
> [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) 
> [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_91]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ProjectMergeRule:force_mode, args 
> [rel#5:LogicalProject.NONE.ANY([]).[](input=rel#11103:Subset#6.NONE.ANY([]).[],state_nm='13'),
>  
> rel#11107:LogicalProject.NONE.ANY([]).[](input=rel#11095:Subset#4.NONE.ANY([]).[],state=$1)]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ProjectMergeRule:force_mode, args 
> [rel#5:LogicalProject.NONE.ANY([]).[](input=rel#11103:Subset#6.NONE.ANY([]).[],state_nm='13'),
>  
> rel#11107:LogicalProject.NONE.ANY([]).[](input=rel#11095:Subset#4.NONE.ANY([]).[],state=$1)]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303) 
> ~[calcite-core-1.4.0-drill-r17.jar:1.4.0-drill-r17]
> at 
> 

[jira] [Created] (DRILL-5048) AssertionError when case statement is used with timestamp and null

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5048:


 Summary: AssertionError when case statement is used with timestamp 
and null
 Key: DRILL-5048
 URL: https://issues.apache.org/jira/browse/DRILL-5048
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.9.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: Future


AssertionError when we use case with timestamp and null:

{noformat}
0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END
. . . . . . . . . . . . . . > FROM
. . . . . . . . . . . . . . > (
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE 
CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res
. . . . . . . . . . . . . . > FROM (values(1)) foo
. . . . . . . . . . . . . . > ) foobar;
Error: SYSTEM ERROR: AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL


[Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] 
(state=,code=0)
{noformat}

Stack trace from drillbit.log

{noformat}
Caused by: java.lang.AssertionError: Type mismatch:
rowtype of new rel:
RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL
rowtype of set:
RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL
at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) 
~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138)
 ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18]
... 16 common frames omitted
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673402#comment-15673402
 ] 

Serhii Harnyk commented on DRILL-5032:
--

List of columns in every hive partition increases size of serialized physical 
plan that causes OOM.
As Hive allows every partition to have its own set of columns now system 
checks, and if all column sets for all partitions are equal, list of columns 
stores only on hive table level.

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
> ~[protobuf-java-2.5.0.jar:na]
> at 
> com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
> ~[protobuf-java-2.5.0.jar:na]
> at 

[jira] [Commented] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654543#comment-15654543
 ] 

Serhii Harnyk commented on DRILL-5032:
--

As [~jni] mentioned

Today, the physical plan looks like:
listOfColumns : [col1, col2, ...] — TableLevel
Partitons : [
partiton1 :
{ listOfColums : [col1, col2, ...] -- PartitonLevel  }
,
partiton2 :
{ listOfColums : [col1, col2, ...] -- PartitonLevel  }
,
... 
partiton_n :
{ listOfColums : [col1, col2, ...] -- PartitonLevel  }
,
]
The listOfColumns are repeating in every partition, which seems to be 
unnecessary. We should get rid of those repeated list of columns in each 
partition, as long as they are same as the listOfColumns at Table level.

So the initial idea is to remove repeated listOfColums from HivePartition 
physical plan serialization 

> Drill query on hive parquet table failed with OutOfMemoryError: Java heap 
> space
> ---
>
> Key: DRILL-5032
> URL: https://issues.apache.org/jira/browse/DRILL-5032
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Hive
>Affects Versions: 1.8.0
>Reporter: Serhii Harnyk
>Assignee: Serhii Harnyk
>
> Following query on hive parquet table failed with OOM Java heap space:
> {code}
> select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
> vmdr_trades where trade_date='2016-04-12'
> 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 1 ms
> 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 3 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
> class: 
> org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
> filter tree: 0 ms
> 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
> o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
> partition pruning.Total pruning elapsed time: 0 ms
> 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
> o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, 
> exiting. Information message: Unable to handle out of memory condition in 
> Foreman.
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>  ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:136) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:166) 
> ~[na:1.8.0_74]
> at java.lang.StringBuilder.append(StringBuilder.java:76) 
> ~[na:1.8.0_74]
> at 
> com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
> ~[protobuf-java-2.5.0.jar:na]
> 

[jira] [Created] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-5032:


 Summary: Drill query on hive parquet table failed with 
OutOfMemoryError: Java heap space
 Key: DRILL-5032
 URL: https://issues.apache.org/jira/browse/DRILL-5032
 Project: Apache Drill
  Issue Type: Bug
  Components: Functions - Hive
Affects Versions: 1.8.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk


Following query on hive parquet table failed with OOM Java heap space:
{code}
select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from 
vmdr_trades where trade_date='2016-04-12'
2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 1 ms
2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 3 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning 
class: 
org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze 
filter tree: 0 ms
2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO  
o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for 
partition pruning.Total pruning elapsed time: 0 ms
2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR 
o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. 
Information message: Unable to handle out of memory condition in Foreman.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) 
~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
 ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) 
~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:166) ~[na:1.8.0_74]
at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74]
at 
com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.print(TextFormat.java:71) 
~[protobuf-java-2.5.0.jar:na]
at com.google.protobuf.TextFormat.printToString(TextFormat.java:118) 
~[protobuf-java-2.5.0.jar:na]
at 
com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:106) 
~[protobuf-java-2.5.0.jar:na]
at 

[jira] [Commented] (DRILL-4944) incorrect results - case expression

[Serhii Harnyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622344#comment-15622344
 ] 

Serhii Harnyk commented on DRILL-4944:
--

Result of SELECT 0.1 FROM (values(1)) where 1=1

1 row(s):
-
| EXPR$0
-
| 0.1 
-

Result of SELECT CAST(0.1 as float) FROM (values(1)) where 1=1
-
| EXPR$0  |
-
| 0.1   |
-

As you can see default type for 0.1 is FLOAT8 - "double", not "float". 
So the query like SELECT 1 FROM (values(1)) where 0.1 = CAST(0.1 as float)
will have a result:
0 row(s):
Total record count: 0

The problem is that Drill when does comparison considers type of argument, not 
only value. So we need to decide - is it bug or feature.

There are two ways for fixing this "bug".
1. Set default type floats - float4.
2. Don't consider type in comparisons. 
Both ways can cause huge effects for the system.

> incorrect results - case expression
> ---
>
> Key: DRILL-4944
> URL: https://issues.apache.org/jira/browse/DRILL-4944
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.9.0
>Reporter: Khurram Faraaz
>Assignee: Serhii Harnyk
>Priority: Critical
> Fix For: 1.9.0
>
>
> Drill 1.9.0 (git commit id: 4edabe7a) returns null, which is wrong.
>  {noformat}
>  0: jdbc:drill:schema=dfs.tmp> SELECT res2, case res2 WHEN 0.1 THEN 0. 
> ELSE null END
> . . . . . . . . . . . . . . > FROM
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE CAST(0.1 
> as float) end) res2
> . . . . . . . . . . . . . . > FROM (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar ;
> +---+-+
> | res2  | EXPR$1  |
> +---+-+
> | 0.1   | null|
> +---+-+
> 1 row selected (0.106 seconds)
>  {noformat}
>  
>  Postgres returns correct results
>   {noformat}
>  postgres=# SELECT res2, case res2 WHEN 0.1 THEN 0. ELSE null END
> postgres-# FROM
> postgres-# (
> postgres(# SELECT
> postgres(# (CASE WHEN (false) THEN null ELSE CAST(0.1 as float) end) res2
> postgres(# FROM (values(1)) foo
> postgres(# ) foobar ;
>  res2 |  case
> --+
>   0.1 | 0.
> (1 row)
>  {noformat}
>  
>  Calcite also returns correct results
>   {noformat}
>  0: jdbc:calcite:model=target/test-classes/mod> SELECT res2, case res2 WHEN 
> 0.1 THEN 0. ELSE null END
> . . . . . . . . . . . . . . . . . . . . . . .> FROM
> . . . . . . . . . . . . . . . . . . . . . . .> (
> . . . . . . . . . . . . . . . . . . . . . . .>   SELECT
> . . . . . . . . . . . . . . . . . . . . . . .>  (CASE WHEN (false) 
> THEN null ELSE CAST(0.1 as float) end) res2
> . . . . . . . . . . . . . . . . . . . . . . .>  FROM (values(1)) foo
> . . . . . . . . . . . . . . . . . . . . . . .> ) foobar ;
> +-++
> |  RES2   | EXPR$1 |
> +-++
> | 0.1 | 0. |
> +-++
> 1 row selected (1.277 seconds)
>  {noformat}
>  
>  Details of explain plan from Drill 1.9.0
>   {noformat}
>  0: jdbc:drill:schema=dfs.tmp> explain plan for SELECT res2, case res2 WHEN 
> 0.1 THEN 0. ELSE null END
> . . . . . . . . . . . . . . > FROM
> . . . . . . . . . . . . . . > (
> . . . . . . . . . . . . . . >   SELECT
> . . . . . . . . . . . . . . >  (CASE WHEN (false) THEN null ELSE 
> CAST(0.1 as float) end) res2
> . . . . . . . . . . . . . . >  FROM (values(1)) foo
> . . . . . . . . . . . . . . > ) foobar ;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01  Project(res2=[$0], EXPR$1=[$1])
> 00-02Project(res2=[CASE(false, null, 0.1)], 
> EXPR$1=[CASE(=(CASE(false, null, 0.1), 0.1), 0., null)])
> 00-03  Values
>  | {
>   "head" : {
> "version" : 1,
> "generator" : {
>   "type" : "ExplainHandler",
>   "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ ],
> "queue" : 0,
> "resultMode" : "EXEC"
>   },
>   "graph" : [ {
> "pop" : "Values",
> "@id" : 3,
> "content" : [ {
>   "EXPR$0" : {
> "$numberLong" : 1
>   }
> } ],
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 1.0
>   }, {
> "pop" : "project",
> "@id" : 2,
> "exprs" : [ {
>   "ref" : "`res2`",
>   

[jira] [Created] (DRILL-4906) CASE Expression with constant generates class exception

[Serhii Harnyk (JIRA)]

Serhii Harnyk created DRILL-4906:


 Summary: CASE Expression with constant generates class exception
 Key: DRILL-4906
 URL: https://issues.apache.org/jira/browse/DRILL-4906
 Project: Apache Drill
  Issue Type: Bug
  Components: SQL Parser
Affects Versions: 1.8.0, 1.6.0
Reporter: Serhii Harnyk
Assignee: Serhii Harnyk
 Fix For: 1.9.0


How to reproduce:

select (case when (true) then 1 end) from (values(1));

Error
Error: SYSTEM ERROR: ClassCastException: 
org.apache.drill.exec.expr.holders.NullableVarCharHolder cannot be cast to 
org.apache.drill.exec.expr.holders.VarCharHolder



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)