[jira] [Created] (DRILL-6256) Remove references to java 7 from readme and other files

[Volodymyr Tkach (JIRA)]

Volodymyr Tkach created DRILL-6256:
--

 Summary: Remove references to java 7 from readme and other files
 Key: DRILL-6256
 URL: https://issues.apache.org/jira/browse/DRILL-6256
 Project: Apache Drill
  Issue Type: Bug
Reporter: Volodymyr Tkach
Assignee: Volodymyr Tkach


Since master branch uses jdk 8 we should remove all references to java 7.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401470#comment-16401470
 ] 

Kunal Khatua commented on DRILL-6242:
-

We might be able to go with option C. 

Quoting this from http://www.joda.org/joda-time/
{quote}The standard date and time classes prior to Java SE 8 are poor. By 
tackling this problem head-on, Joda-Time became the de facto standard date and 
time library for Java prior to Java SE 8. *Note that from Java SE 8 onwards, 
users are asked to migrate to java.time (JSR-310) - a core part of the JDK 
which replaces this project.*{quote}

Correct me if I'm wrong, but I'm guessing that would solve your problem, as we 
could move to {{java.sql.*}} formats. However, this essentially amounts to a 
non-trivial amount of code refactoring.

> Output format for nested date, time, timestamp values in an object hierarchy
> 
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that 
> contain nested fields of date, time, timestamp types.  When a query returns 
> these objects, the output format for the nested date, time, timestamp, are 
> showing the internal object (org.joda.time.DateTime), rather than the logical 
> data value.
> For example.  Suppose in MongoDB, we have a single object that looks like 
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query 
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t; 
> ++-+ 
> | EXPR$0 | context | 
> ++-+ 
> | 2018-03-13 | 
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfDay":1312,"hourOfDay":21,"zone":{"fixed":true,"id":"UTC"},"millis":1520977974940,"chronology":{"zone":{"fixed":true,"id":"UTC"}},"afterNow":false,"beforeNow":true,"equalNow":false},"user":"jack"}
>  |
> {code}
> We can see that from the above output, when the date field is retrieved as a 
> top level column, Drill outputs a logical date value.  But when the same 
> field is within an object hierarchy, Drill outputs the internal object used 
> to hold the date value.
> The expected output is the same display for whether the date field is shown 
> as a top level column or when it is within an object hierarchy:
> {code:java}
> > select t.context.`date`, t.context from test t; 
> ++-+ 
> | EXPR$0 | context | 
> ++-+ 
> | 2018-03-13 | {"date":"2018-03-13","user":"jack"} |
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6145) Implement Hive MapR-DB JSON handler.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401464#comment-16401464
 ] 

ASF GitHub Bot commented on DRILL-6145:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/1158
  
@vdiravka please do link the JIRA that you will create.
@vrozov can you take a final look?


> Implement Hive MapR-DB JSON handler. 
> -
>
> Key: DRILL-6145
> URL: https://issues.apache.org/jira/browse/DRILL-6145
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive, Storage - MapRDB
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.14.0
>
>
> Similar to "hive-hbase-storage-handler" to support querying MapR-DB Hive's 
> external tables it is necessary to add "hive-maprdb-json-handler".
> Use case:
>  # Create a table MapR-DB JSON table:
> {code}
> _> mapr dbshell_
> _maprdb root:> create /tmp/table/json_  (make sure /tmp/table exists)
> {code}
> -- insert data
> {code}
> insert /tmp/table/json --value '\{"_id":"movie002" , "title":"Developers 
> on the Edge", "studio":"Command Line Studios"}'
> insert /tmp/table/json --id movie003 --value '\{"title":"The Golden 
> Master", "studio":"All-Nighter"}'
> {code} 
>  #  Create a Hive external table:
> {code}
> hive> CREATE EXTERNAL TABLE mapr_db_json_hive_tbl ( 
> > movie_id string, title string, studio string) 
> > STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler' 
> > TBLPROPERTIES("maprdb.table.name" = 
> "/tmp/table/json","maprdb.column.id" = "movie_id");
> {code}
>  
>  #  Use hive schema to query this table via Drill:
> {code}
> 0: jdbc:drill:> select * from hive.mapr_db_json_hive_tbl;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6231) Fix memory allocation for repeated list vector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401461#comment-16401461
 ] 

ASF GitHub Bot commented on DRILL-6231:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/1171
  
@paul-rogers please review.


> Fix memory allocation for repeated list vector
> --
>
> Key: DRILL-6231
> URL: https://issues.apache.org/jira/browse/DRILL-6231
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
> Fix For: 1.14.0
>
>
> Vector allocation in record batch sizer can be enhanced to allocate memory 
> for repeated list vector more accurately rather than using default functions. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6230) Extend row set readers to handle hyper vectors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401460#comment-16401460
 ] 

ASF GitHub Bot commented on DRILL-6230:
---

Github user ppadma commented on the issue:

https://github.com/apache/drill/pull/1161
  
@paul-rogers I started the review. will get back soon.


> Extend row set readers to handle hyper vectors
> --
>
> Key: DRILL-6230
> URL: https://issues.apache.org/jira/browse/DRILL-6230
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.14.0
>
>
> The current row set readers have incomplete support for hyper-vectors. To add 
> full support, we need an interface that supports either single batches or 
> hyper batches. Accessing vectors in hyper batches differs depending on 
> whether the vector is at the top level or is nested. See [this 
> post|https://github.com/paul-rogers/drill/wiki/BH-Column-Readers] for 
> details. Also includes a simpler reader template: replaces the original three 
> classes with one, in parallel with the writers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-520) ceiling/ceil and floor functions return decimal value instead of an integer

[Karthikeyan Manivannan (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401459#comment-16401459
 ] 

Karthikeyan Manivannan commented on DRILL-520:
--

This cannot be fixed simply by changing the return type of the builtin 
functions (as suggested in the attached patch) because floating point types 
(Float4, Float8) have a range that is beyond what can be represented in any 
integer type, even in BigInt.

This is what happens if the patch (change the outputType of ceil/floor to Int 
and use a (int) cast to return the value of java.lang.Math.ceil/floor) is 
applied

0: jdbc:drill:zk=local> select floor(cast('34028.5' as float));

 

| 34028  |

1 row selected (0.157 seconds)

0: jdbc:drill:zk=local> select floor(cast('340280.5' as float));   <--- too 
big to represent as Int

| 2147483647  |  <- Int max

A similar overflow will happen even if a BigInt output type and a 'double' cast 
is used.

I looked up how ProsgreSQL implements ceil/floor:
|Name|Return Type|Description|Example|Result|
|{{ceil(dp}} or {{numeric}})|same as input)|nearest integer greater than or 
equal to argument|{{ceil(-42.8)}}|{{-42}}|
|{{floor(dp}} or {{numeric}})|(same as input)|nearest integer less than or 
equal to argument|{{floor(-42.8)}}|{{-43}}|

where 

_*dp*_ is a double precision 8byte floating-point number
 _*numeric*_ is numeric [ (p, s) ] exact numeric of selectable precision

I am not sure how a the dp/numeric return type is returned as an Int

I think the way to fix this in Drill would be to use BigDecimal as the return 
type. 

 

 

> ceiling/ceil and floor functions return decimal value instead of an integer
> ---
>
> Key: DRILL-520
> URL: https://issues.apache.org/jira/browse/DRILL-520
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.0.0
>Reporter: Krystal
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Fix For: Future
>
> Attachments: DRILL-520.patch
>
>
> Ran the following queries in drill:
> 0: jdbc:drill:schema=dfs> select ceiling(55.8) from dfs.`student` where 
> rownum=11;
> ++
> |   EXPR$0   |
> ++
> | 56.0   |
> ++
> 0: jdbc:drill:schema=dfs> select floor(55.8) from dfs.`student` where 
> rownum=11;
> ++
> |   EXPR$0   |
> ++
> | 55.0   |
> ++
> The same queries executed from oracle, postgres and mysql returned integer 
> values of 56 and 55.
> Found the following description of the two functions from 
> http://users.atw.hu/sqlnut/sqlnut2-chp-4-sect-4.html :
> Ceil/Ceiling:
> Rounds a noninteger value upwards to the next greatest integer. Returns an 
> integer value unchanged.
> Floor:
> Rounds a noninteger value downwards to the next least integer. Returns an 
> integer value unchanged.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-520) ceiling/ceil and floor functions return decimal value instead of an integer

[Karthikeyan Manivannan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthikeyan Manivannan resolved DRILL-520.
--
Resolution: Later

> ceiling/ceil and floor functions return decimal value instead of an integer
> ---
>
> Key: DRILL-520
> URL: https://issues.apache.org/jira/browse/DRILL-520
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.0.0
>Reporter: Krystal
>Assignee: Karthikeyan Manivannan
>Priority: Critical
> Fix For: Future
>
> Attachments: DRILL-520.patch
>
>
> Ran the following queries in drill:
> 0: jdbc:drill:schema=dfs> select ceiling(55.8) from dfs.`student` where 
> rownum=11;
> ++
> |   EXPR$0   |
> ++
> | 56.0   |
> ++
> 0: jdbc:drill:schema=dfs> select floor(55.8) from dfs.`student` where 
> rownum=11;
> ++
> |   EXPR$0   |
> ++
> | 55.0   |
> ++
> The same queries executed from oracle, postgres and mysql returned integer 
> values of 56 and 55.
> Found the following description of the two functions from 
> http://users.atw.hu/sqlnut/sqlnut2-chp-4-sect-4.html :
> Ceil/Ceiling:
> Rounds a noninteger value upwards to the next greatest integer. Returns an 
> integer value unchanged.
> Floor:
> Rounds a noninteger value downwards to the next least integer. Returns an 
> integer value unchanged.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6231) Fix memory allocation for repeated list vector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401456#comment-16401456
 ] 

ASF GitHub Bot commented on DRILL-6231:
---

GitHub user ppadma opened a pull request:

https://github.com/apache/drill/pull/1171

DRILL-6231: Fix memory allocation for repeated list vector



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppadma/drill DRILL-6231

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1171.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1171


commit 114a526240a272bd626862a02d1acd06b9c8b4cf
Author: Padma Penumarthy 
Date:   2018-03-16T04:50:54Z

DRILL-6231: Fix memory allocation for repeated list vector




> Fix memory allocation for repeated list vector
> --
>
> Key: DRILL-6231
> URL: https://issues.apache.org/jira/browse/DRILL-6231
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
> Fix For: 1.14.0
>
>
> Vector allocation in record batch sizer can be enhanced to allocate memory 
> for repeated list vector more accurately rather than using default functions. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401444#comment-16401444
 ] 

ASF GitHub Bot commented on DRILL-6016:
---

Github user rajrahul commented on the issue:

https://github.com/apache/drill/pull/1166
  
The schema given below creates the issue, as @vdiravka pointed int96 is 
marked required here. This parquet was generated with an older version of spark 
and is included in the test case.

```
message spark_schema {
  optional binary article_no (UTF8);
  optional binary qty (UTF8);
  required int96 run_date;
}

```
Newer spark version created the schema below where int96 has become 
optional.

```
message spark_schema {
  optional binary country (UTF8);
  optional double sales;
  optional int96 targetDate;
}
```


> Error reading INT96 created by Apache Spark
> ---
>
> Key: DRILL-6016
> URL: https://issues.apache.org/jira/browse/DRILL-6016
> Project: Apache Drill
>  Issue Type: Bug
> Environment: Drill 1.11
>Reporter: Rahul Raj
>Priority: Major
> Fix For: 1.14.0
>
>
> Hi,
> I am getting the error - SYSTEM ERROR : ClassCastException: 
> org.apache.drill.exec.vector.TimeStampVector cannot be cast to 
> org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark 
> INT96 datetime field on Drill 1.11 in spite of setting the property 
> store.parquet.reader.int96_as_timestamp to  true.
> I believe this was fixed in drill 
> 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong.
> I have attached the dataset at 
> https://github.com/rajrahul/files/blob/master/result.tar.gz



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6231) Fix memory allocation for repeated list vector

[Padma Penumarthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Padma Penumarthy updated DRILL-6231:

Description: Vector allocation in record batch sizer can be enhanced to 
allocate memory for repeated list vector more accurately rather than using 
default functions.   (was: Helper routines in record batch sizer and 
AllocationHelper can be enhanced to allocate memory for repeated list vector 
more accurately rather than using default functions. )

> Fix memory allocation for repeated list vector
> --
>
> Key: DRILL-6231
> URL: https://issues.apache.org/jira/browse/DRILL-6231
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
> Fix For: 1.14.0
>
>
> Vector allocation in record batch sizer can be enhanced to allocate memory 
> for repeated list vector more accurately rather than using default functions. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401341#comment-16401341
 ] 

Kunal Khatua commented on DRILL-6246:
-

[~parthc] would this be a release blocker?

> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401292#comment-16401292
 ] 

Vlad Rozov commented on DRILL-6252:
---

[~vdonapati] Please provide drillbit logs with debug level enabled.

> Foreman node is going down when the non foreman node is stopped
> ---
>
> Key: DRILL-6252
> URL: https://issues.apache.org/jira/browse/DRILL-6252
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Venkata Jyothsna Donapati
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> Two drillbits are running. I'm running a join query over parquet and tried to 
> stop the non-foreman node using drillbit.sh stop. The query fails with 
> *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The 
> non-foreman node goes down. The foreman node also goes down. When I looked at 
> the drillbit.log of both foreman and non-foreman I found that there is memory 
> leak  "Memory was leaked by query. Memory leaked: 
> (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
> 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
> the stack traces for memory leaks 
> {noformat} 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
>  
> Fragment 2:1 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
>         at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
> {noformat} 
>  
> Ping me for the logs and more information.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated DRILL-6252:
--
Description: 
Two drillbits are running. I'm running a join query over parquet and tried to 
stop the non-foreman node using drillbit.sh stop. The query fails with *"Error: 
DATA_READ ERROR: Exception occurred while reading from disk".* The non-foreman 
node goes down. The foreman node also goes down. When I looked at the 
drillbit.log of both foreman and non-foreman I found that there is memory leak  
"Memory was leaked by query. Memory leaked: 
(2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
the stack traces for memory leaks 

{noformat} 
[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory 
leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)
{noformat} 

 

Ping me for the logs and more information.

 

  was:
Two drillbits are running. I'm running a join query over parquet and tried to 
stop the non-foreman node using drillbit.sh stop. The query fails with *"Error: 
DATA_READ ERROR: Exception occurred while reading from disk".* The non-foreman 
node goes down. The foreman node also goes down. When I looked at the 
drillbit.log of both foreman and non-foreman I found that there is memory leak  
"Memory was leaked by query. Memory leaked: 
(2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
the stack traces for memory leaks 

 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory 
leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)

 

Ping me for the logs and more information.

 


> Foreman node is going down when the non foreman node is stopped
> 

[jira] [Commented] (DRILL-4091) Support more functions in gis contrib module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401212#comment-16401212
 ] 

ASF GitHub Bot commented on DRILL-4091:
---

Github user brendanstennett commented on the issue:

https://github.com/apache/drill/pull/258
  
Hi @cgivre .  I have some spare cycles coming up in our next open sprint 
starting Apr 2 that either I or someone on my team can take a look at it to get 
it across if it isn't already.  Will keep you updated.  What is your timeline 
for the next release?


> Support more functions in gis contrib module
> 
>
> Key: DRILL-4091
> URL: https://issues.apache.org/jira/browse/DRILL-4091
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>Reporter: Karol Potocki
>Assignee: Karol Potocki
>Priority: Major
>
> Support for commonly used gis functions in gis contrib module: relate, 
> contains, crosses, intersects, touches, difference, disjoint, buffer, union 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6223) Drill fails on Schema changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401197#comment-16401197
 ] 

ASF GitHub Bot commented on DRILL-6223:
---

Github user sachouche commented on the issue:

https://github.com/apache/drill/pull/1170
  
@amansinha100 can you please review this pull request?

Thanks!


> Drill fails on Schema changes 
> --
>
> Key: DRILL-6223
> URL: https://issues.apache.org/jira/browse/DRILL-6223
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0, 1.12.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> Drill Query Failing when selecting all columns from a Complex Nested Data 
> File (Parquet) Set). There are differences in Schema among the files:
>  * The Parquet files exhibit differences both at the first level and within 
> nested data types
>  * A select * will not cause an exception but using a limit clause will
>  * Note also this issue seems to happen only when multiple Drillbit minor 
> fragments are involved (concurrency higher than one)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6223) Drill fails on Schema changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401193#comment-16401193
 ] 

ASF GitHub Bot commented on DRILL-6223:
---

GitHub user sachouche opened a pull request:

https://github.com/apache/drill/pull/1170

DRILL-6223: Fixed several Drillbit failures due to schema changes

Fixed several Issues due to Schema changes:
1) Changes in complex data types
Drill Query Failing when selecting all columns from a Complex Nested Data 
File (Parquet) Set). There are differences in Schema among the files:

The Parquet files exhibit differences both at the first level and within 
nested data types
A select * will not cause an exception but using a limit clause will
Note also this issue seems to happen only when multiple Drillbit minor 
fragments are involved (concurrency higher than one)

2) Dangling columns (both simple and complex)
This situation can be easily reproduced for:
- Select STAR queries which involve input data with different schemas
- LIMIT or / and PROJECT operators are used
- The data will be read from more than one minor fragment
- This is because individual readers have logic to handle such use-cases 
but not downstream operators
- So is reader-1 sends one batch with F1, F2, and F3
- The reader-2 sends batch F2, F3
- Then the LIMIT and PROJECT operator will fail to cleanup the dangling 
column F1 which will cause failures when downstream operators copy logic 
attempts copy the stale column F1
- This pull request adds logic to detect and eliminate dangling columns   

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sachouche/drill DRILL-6223

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1170


commit d986b6c7588c107bb7e49d2fc8eb3f25a60e1214
Author: Salim Achouche 
Date:   2018-02-21T02:17:14Z

DRILL-6223: Fixed several Drillbit failures due to schema changes




> Drill fails on Schema changes 
> --
>
> Key: DRILL-6223
> URL: https://issues.apache.org/jira/browse/DRILL-6223
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Relational Operators
>Affects Versions: 1.10.0, 1.12.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> Drill Query Failing when selecting all columns from a Complex Nested Data 
> File (Parquet) Set). There are differences in Schema among the files:
>  * The Parquet files exhibit differences both at the first level and within 
> nested data types
>  * A select * will not cause an exception but using a limit clause will
>  * Note also this issue seems to happen only when multiple Drillbit minor 
> fragments are involved (concurrency higher than one)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally

[Sorabh Hamirwasia (JIRA)]

Sorabh Hamirwasia created DRILL-6255:


 Summary: Drillbit while sending control message to itself creates 
a connection instead of submitting locally
 Key: DRILL-6255
 URL: https://issues.apache.org/jira/browse/DRILL-6255
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
 Fix For: 1.14.0


With the new shutdown feature introduced in 1.12, there is a state introduced 
in DrillbitEndpoint. Due to this the equality check happening 
[here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
 will result in false and hence the fragments supposed to be scheduled on 
Foreman will be treated as remote fragments and a connection will be created to 
schedule it. The equality check is false because localEndpoint state is STARTUP 
whereas state in assigned Drillbit is ONLINE.

I guess now we should update the equality check to verify just for address and 
control port to be same between assigned and local Drillbit endpoint. A test 
can be added for this based on _sys.memory_ table since that guarantees 
scheduling minor fragments on each Drillbit node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-6254:
-

 Summary: IllegalArgumentException: the requested size must be 
non-negative
 Key: DRILL-6254
 URL: https://issues.apache.org/jira/browse/DRILL-6254
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
 Attachments: genAllTypesJSN.py

Flatten query fails due to IllegalArgumentException: the requested size must be 
non-negative.

Script to generate JSON data file is attached here.

{noformat}
0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS 
. . . . . . . . . . . . . . > SELECT
. . . . . . . . . . . . . . > CAST( col_int AS INT) col_int, 
. . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint, 
. . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char, 
. . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256)) 
col_fxdln_str, 
. . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256)) 
col_varln_str, 
. . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float, 
. . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION) col_double, 
. . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date, 
. . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time, 
. . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp, 
. . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean, 
. . . . . . . . . . . . . . > col_binary, 
. . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 9 |
+---++
1 row selected (0.29 seconds)
{noformat}

Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;

output_batch_size was set to its default value
drill.exec.memory.operator.output_batch_size = 16777216

 

{noformat}
select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;

Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be 
non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010] 
(state=,code=0)
0: jdbc:drill:schema=dfs.tmp>
{noformat}

Stack trace from drillbit.log

{noformat}
2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from 
tbl_all_types_jsn_to_parquet
2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 
using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of 1 
using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs, 
Average start: 2.829000 μs .
2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO 
o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.f.FragmentStatusReporter - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State to report: RUNNING
2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested RUNNING --> FAILED
2018-03-15 12:19:44,927 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO 
o.a.d.e.w.fragment.FragmentExecutor - 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: 
State change requested FAILED --> FINISHED
2018-03-15 12:19:44,928 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalArgumentException: 
the requested size must be non-negative

Fragment 0:0

[Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 

[jira] [Created] (DRILL-6253) HashAgg Unit Testing And Refactoring

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-6253:
-

 Summary: HashAgg Unit Testing And Refactoring
 Key: DRILL-6253
 URL: https://issues.apache.org/jira/browse/DRILL-6253
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Timothy Farkas
Assignee: Timothy Farkas


This is a parent issue to hold all the subtasks required to refactor HashAgg to 
make it unit testable. Design doc

https://docs.google.com/document/d/110BAWg3QXMfdmuqB0p3HuaoKpPGY-lqCRtHFxdh53Ds/edit?usp=sharing





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6243) Alert box to confirm shutdown of drillbit after clicking shutdown button

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401100#comment-16401100
 ] 

ASF GitHub Bot commented on DRILL-6243:
---

Github user dvjyothsna commented on the issue:

https://github.com/apache/drill/pull/1169
  
@sohami Please review


> Alert box to confirm shutdown of drillbit after clicking shutdown button 
> -
>
> Key: DRILL-6243
> URL: https://issues.apache.org/jira/browse/DRILL-6243
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Venkata Jyothsna Donapati
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6243) Alert box to confirm shutdown of drillbit after clicking shutdown button

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401098#comment-16401098
 ] 

ASF GitHub Bot commented on DRILL-6243:
---

GitHub user dvjyothsna opened a pull request:

https://github.com/apache/drill/pull/1169

DRILL-6243: Added alert box to confirm shutdown of drillbit after cli…

…cking shutdown button

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dvjyothsna/drill DRILL-6243

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1169.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1169


commit 9b4b0700937e9d155324ebbd119171588dcf97c1
Author: dvjyothsna 
Date:   2018-03-14T23:36:42Z

DRILL-6243: Added alert box to confirm shutdown of drillbit after clicking 
shutdown button




> Alert box to confirm shutdown of drillbit after clicking shutdown button 
> -
>
> Key: DRILL-6243
> URL: https://issues.apache.org/jira/browse/DRILL-6243
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Venkata Jyothsna Donapati
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-6252:


Assignee: Vlad Rozov

> Foreman node is going down when the non foreman node is stopped
> ---
>
> Key: DRILL-6252
> URL: https://issues.apache.org/jira/browse/DRILL-6252
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Venkata Jyothsna Donapati
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> Two drillbits are running. I'm running a join query over parquet and tried to 
> stop the non-foreman node using drillbit.sh stop. The query fails with 
> *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The 
> non-foreman node goes down. The foreman node also goes down. When I looked at 
> the drillbit.log of both foreman and non-foreman I found that there is memory 
> leak  "Memory was leaked by query. Memory leaked: 
> (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
> 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
> the stack traces for memory leaks 
>  
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
>  
> Fragment 2:1 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
>         at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
> Ping me for the logs and more information.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6252:
-
Fix Version/s: 1.14.0

> Foreman node is going down when the non foreman node is stopped
> ---
>
> Key: DRILL-6252
> URL: https://issues.apache.org/jira/browse/DRILL-6252
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Venkata Jyothsna Donapati
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> Two drillbits are running. I'm running a join query over parquet and tried to 
> stop the non-foreman node using drillbit.sh stop. The query fails with 
> *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The 
> non-foreman node goes down. The foreman node also goes down. When I looked at 
> the drillbit.log of both foreman and non-foreman I found that there is memory 
> leak  "Memory was leaked by query. Memory leaked: 
> (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
> 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
> the stack traces for memory leaks 
>  
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
>  
> Fragment 2:1 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
>         at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
> Ping me for the logs and more information.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-6252:
--
Priority: Critical  (was: Major)

> Foreman node is going down when the non foreman node is stopped
> ---
>
> Key: DRILL-6252
> URL: https://issues.apache.org/jira/browse/DRILL-6252
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Venkata Jyothsna Donapati
>Priority: Critical
>
> Two drillbits are running. I'm running a join query over parquet and tried to 
> stop the non-foreman node using drillbit.sh stop. The query fails with 
> *"Error: DATA_READ ERROR: Exception occurred while reading from disk".* The 
> non-foreman node goes down. The foreman node also goes down. When I looked at 
> the drillbit.log of both foreman and non-foreman I found that there is memory 
> leak  "Memory was leaked by query. Memory leaked: 
> (2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
> 100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
> the stack traces for memory leaks 
>  
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
>  
> Fragment 2:1 
> [Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]
>         at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (3145728)
> Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
> (res/actual/peak/limit)
>  
> Ping me for the logs and more information.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6251) Queries from system tables are hang

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-6251:
---
Description: 
On CentoOS cluster Drill hangs while querying sys tables after "use dfs;" 
(embedded or distributed mode):
{code}
0: jdbc:drill:> select * from sys.version;
+--+---+--++++
| version  | commit_id |
commit_message|commit_time |build_email 
| build_time |
+--+---+--++++
| 1.13.0   | 796fcf051b3553c4597abbdca5ca247b139734ba  | [maven-release-plugin] 
prepare release drill-1.13.0  | 13.03.2018 @ 11:39:14 IST  | par...@apache.org  
| 13.03.2018 @ 13:13:45 IST  |
+--+---+--++++
1 row selected (3.784 seconds)
0: jdbc:drill:> use dfs;
+---+--+
|  ok   | summary  |
+---+--+
| true  | Default schema changed to [dfs]  |
+---+--+
1 row selected (0.328 seconds)
0: jdbc:drill:> select * from sys.version;
Error: Statement canceled (state=,code=0)
0: jdbc:drill:> 
{code}
*Note*: there is no failure on local debian machine with Drill in embedded mode.
dfs pugin configs are default (with "connection": "file:///", other file 
systems works good).

This failure is connected to DRILL-5089 and Calcite rebase. 
Related commits:
https://github.com/apache/drill/commit/450e67094eb6e9a6484d7f86c49b51c77a08d7b2
https://github.com/apache/drill/commit/18a71a38f6bd1fd33d21d1c68fc23c5901b0080a

After analyzing in remote debug I found the following flow:
"dfs" DynamicRootSchema is created, then a new "sys" one is created.
After Calcite validate "sys" SimpleCalciteSchema is created. But in 
WorkspaceSchemaFactory#create  wrong WorkspaceConfig is left and "/" is 
combined with "sys".

  was:
On CentoOS cluster Drill hangs while querying sys tables after "use dfs;" 
(embedded or distributed mode):
{code}
0: jdbc:drill:> select * from sys.version;
+--+---+--++++
| version  | commit_id |
commit_message|commit_time |build_email 
| build_time |
+--+---+--++++
| 1.13.0   | 796fcf051b3553c4597abbdca5ca247b139734ba  | [maven-release-plugin] 
prepare release drill-1.13.0  | 13.03.2018 @ 11:39:14 IST  | par...@apache.org  
| 13.03.2018 @ 13:13:45 IST  |
+--+---+--++++
1 row selected (3.784 seconds)
0: jdbc:drill:> use dfs;
+---+--+
|  ok   | summary  |
+---+--+
| true  | Default schema changed to [dfs]  |
+---+--+
1 row selected (0.328 seconds)
0: jdbc:drill:> select * from sys.version;
Error: Statement canceled (state=,code=0)
0: jdbc:drill:> 
{code}
*Note*: there is not failure on local debian machine in embedded mode.
dfs pugin configs are default (with "connection": "file:///", other file 
systems works good).

This failure is connected to DRILL-5089 and calcite rebase. 
Related commits:
https://github.com/apache/drill/commit/450e67094eb6e9a6484d7f86c49b51c77a08d7b2
https://github.com/apache/drill/commit/18a71a38f6bd1fd33d21d1c68fc23c5901b0080a

After analyzing in remote debug I found the following flow:
"dfs" DynamicRootSchema is created, then a new "sys" one is created.
After calcite validate "sys" SimpleCalciteSchema is created. But in 
WorkspaceSchemaFactory#create  wrong WorkspaceConfig is left and "/" is 
combined with "sys".


> Queries from system tables are hang
> ---
>
> Key: DRILL-6251
> URL: https://issues.apache.org/jira/browse/DRILL-6251
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>   

[jira] [Assigned] (DRILL-6243) Alert box to confirm shutdown of drillbit after clicking shutdown button

[Venkata Jyothsna Donapati (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Venkata Jyothsna Donapati reassigned DRILL-6243:


Assignee: Venkata Jyothsna Donapati

> Alert box to confirm shutdown of drillbit after clicking shutdown button 
> -
>
> Key: DRILL-6243
> URL: https://issues.apache.org/jira/browse/DRILL-6243
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Venkata Jyothsna Donapati
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4829) Configure the address to bind to

[Venkata Jyothsna Donapati (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401063#comment-16401063
 ] 

Venkata Jyothsna Donapati commented on DRILL-4829:
--

This is incorporated in DRILL-6005. Configuring 
`drill.exec.allow_loopback_address_binding` option to true will allow binding 
to loopback address

> Configure the address to bind to
> 
>
> Key: DRILL-4829
> URL: https://issues.apache.org/jira/browse/DRILL-4829
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Daniel Stockton
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Fix For: 1.14.0
>
>
> 1.7 included the following patch to prevent Drillbits binding to the loopback 
> address: https://issues.apache.org/jira/browse/DRILL-4523
> "Drillbit is disallowed to bind to loopback address in distributed mode."
> It would be better if this was configurable rather than rely on /etc/hosts, 
> since it's common for the hostname to resolve to loopback.
> Would you accept a patch that adds this option to drill.override.conf?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6039) drillbit.sh graceful_stop does not wait for fragments to complete before stopping the drillbit

[Venkata Jyothsna Donapati (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401054#comment-16401054
 ] 

Venkata Jyothsna Donapati commented on DRILL-6039:
--

This happened only when we query on parquet files. And looks like this is the 
issue : https://issues.apache.org/jira/browse/DRILL-6252 and graceful shutdown 
has nothing to do with this 

> drillbit.sh graceful_stop does not wait for fragments to complete before 
> stopping the drillbit
> --
>
> Key: DRILL-6039
> URL: https://issues.apache.org/jira/browse/DRILL-6039
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.3.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> git.commit.id.abbrev=eb0c403
> I have 3-nodes cluster with drillbits running on each node.  I kicked off a 
> long running query.  In the middle of the query, I did a "./drillbit.sh 
> graceful_stop" on one of the non-foreman node.  The node was stopped within a 
> few seconds and the query failed with error:
> Error: SYSTEM ERROR: IOException: Filesystem closed
> Fragment 4:15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401023#comment-16401023
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sohami commented on the issue:

https://github.com/apache/drill/pull/1168
  
+1 LGTM. Thanks for making the changes.


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6215) Use prepared statement instead of Statement in JdbcRecordReader class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400980#comment-16400980
 ] 

ASF GitHub Bot commented on DRILL-6215:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1159
  
@kfaraaz Are there unit tests specific to the JDBC Storage Plugin? They're 
not the same as the JDBC unit tests, which test the Drill JDBC driver.

While the change seems straightforward, I'm not sure if all JDBC drivers 
support PreparedStatement. For e.g., within Drill, the PreparedStatement is 
executed as a standard Statement object, which is why this _works_ functionally.

What happens if a JDBC driver backing the JDBC storage plugin does not 
support PreparedStatement (i.e. it is a No-Op)? 


> Use prepared statement instead of Statement in JdbcRecordReader class
> -
>
> Key: DRILL-6215
> URL: https://issues.apache.org/jira/browse/DRILL-6215
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JDBC
>Affects Versions: 1.12.0
>Reporter: Khurram Faraaz
>Priority: Major
>
> Use prepared statement instead of Statement in JdbcRecordReader class, which 
> is more efficient and less vulnerable to SQL injection attacks.
> Apache Drill 1.13.0-SNAPSHOT, commit : 
> 9073aed67d89e8b2188870d6c812706085c9c41b
> Findbugs reports the below bug and suggests that we use prepared statement 
> instead of Statement.
> {noformat}
> In class org.apache.drill.exec.store.jdbc.JdbcRecordReader
> In method 
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, 
> OutputMutator)
> At JdbcRecordReader.java:[line 170]
> org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(OperatorContext, 
> OutputMutator) passes a nonconstant String to an execute method on an SQL 
> statement
> The method invokes the execute method on an SQL statement with a String that 
> seems to be dynamically generated. 
> Consider using a prepared statement instead. 
> It is more efficient and less vulnerable to SQL injection attacks.
> {noformat}
> LOC - 
> https://github.com/apache/drill/blob/a9ea4ec1c5645ddab4b7aef9ac060ff5f109b696/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcRecordReader.java#L170
> {noformat}
> To run with findbugs:
> mvn clean install -Pfindbugs -DskipTests
> Findbugs will wirite the output to finbugsXml.html in the target directory of 
> each module. 
> For example the java-exec module report is located at: 
> ./exec/java-exec/target/findbugs/findbugsXml.html
> Use 
> find . -name "findbugsXml.html"
> to locate the files.
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6252) Foreman node is going down when the non foreman node is stopped

[Venkata Jyothsna Donapati (JIRA)]

Venkata Jyothsna Donapati created DRILL-6252:


 Summary: Foreman node is going down when the non foreman node is 
stopped
 Key: DRILL-6252
 URL: https://issues.apache.org/jira/browse/DRILL-6252
 Project: Apache Drill
  Issue Type: Bug
Reporter: Venkata Jyothsna Donapati


Two drillbits are running. I'm running a join query over parquet and tried to 
stop the non-foreman node using drillbit.sh stop. The query fails with *"Error: 
DATA_READ ERROR: Exception occurred while reading from disk".* The non-foreman 
node goes down. The foreman node also goes down. When I looked at the 
drillbit.log of both foreman and non-foreman I found that there is memory leak  
"Memory was leaked by query. Memory leaked: 
(2097152)\nAllocator(op:2:0:0:HashPartitionSender) 
100/6291456/6832128/100 (res/actual/peak/limit)\n". Following are 
the stack traces for memory leaks 

 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Memory was leaked by query. Memory leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)

 

 

Fragment 2:1 

[Error Id: 0d9a2799-7e97-46b3-953b-1f8d0dd87a04 on qa102-34.qa.lab:31010]

        at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
 ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:297)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
 [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[na:1.8.0_161]

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[na:1.8.0_161]

        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

Caused by: java.lang.IllegalStateException: Memory was leaked by query. Memory 
leaked: (3145728)

Allocator(op:2:1:0:HashPartitionSender) 100/6291456/6291456/100 
(res/actual/peak/limit)

 

Ping me for the logs and more information.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (DRILL-6111) NullPointerException with Kafka Storage Plugin

[Anton Gozhiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Gozhiy closed DRILL-6111.
---

Kafka fix tested with Drill version 1.13.0-SNAPSHOT (commit id: 
cac2882d5a9e22fbc251e4caf622fe30242ad557)
Cases verified:
 # Original issue wasn't reproduced
 # Submitted physical/logical plan
 # Tried with different queries (filters, agg functions, unions, joins etc)
 # Tried to rerun queries

> NullPointerException with Kafka Storage Plugin
> --
>
> Key: DRILL-6111
> URL: https://issues.apache.org/jira/browse/DRILL-6111
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.12.0
>Reporter: Jared Stehler
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.13.0
>
>
> I'm unable to query using the kafka storage plugin; queries are failing with 
> a NPE which *seems* like a json typo:
> {code:java}
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> Fragment 1:2
> [Error Id: 49d5f72f-0187-480b-8b29-6eeeb5adc88f on 10.80.53.16:31820]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.12.0.jar:1.12.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298)
>  [drill-java-exec-1.12.0.jar:1.12.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.12.0.jar:1.12.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.12.0.jar:1.12.0]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0.jar:1.12.0]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_131]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation 
> of [simple type, class org.apache.drill.exec.store.kafka.KafkaSubScan] value 
> failed (java.lang.NullPointerException): null
> at [Source: {
> "pop" : "single-sender",
> "@id" : 0,
> "receiver-major-fragment" : 0,
> "receiver-minor-fragment" : 0,
> "child" : {
> "pop" : "selection-vector-remover",
> "@id" : 1,
> "child" : {
> "pop" : "limit",
> "@id" : 2,
> "child" : {
> "pop" : "kafka-partition-scan",
> "@id" : 3,
> "userName" : "",
> "columns" : [ "`*`" ],
> "partitionSubScanSpecList" : [ {
> "topicName" : "ingest-prime",
> "partitionId" : 5,
> "startOffset" : 8824294,
> "endOffset" : 8874172
> }, {
> "topicName" : "ingest-prime",
> "partitionId" : 1,
> "startOffset" : 8826346,
> "endOffset" : 8874623
> }, {
> "topicName" : "ingest-prime",
> "partitionId" : 6,
> "startOffset" : 8824744,
> "endOffset" : 8874617
> } ],
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "KafkaStoragePluginConfig" : {
> "type" : "kafka",
> "kafkaConsumerProps" : {
> "key.deserializer" : 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer",
> "auto.offset.reset" : "earliest",
> "bootstrap.servers" : 
> "kafkas.dev3.master.us-west-2.prod.aws.intellify.io:9092",
> "enable.auto.commit" : "true",
> "group.id" : "drill-query-consumer-1",
> "value.deserializer" : 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer",
> "session.timeout.ms" : "3"
> },
> "enabled" : true
> },
> "cost" : 0.0
> },
> "first" : 0,
> "last" : 2,
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> },
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> },
> "destination" : "CgsxMC44MC41My4xNhDM+AEYzfgBIM74ATIGMS4xMi4wOAA=",
> "initialAllocation" : 100,
> "maxAllocation" : 100,
> "cost" : 2.0
> }; line: 49, column: 7] (through reference chain: 
> org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"])
> at 
> com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:263)
>  ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at 
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:453)
>  ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at 
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:472)
>  ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at 
> com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:258)
>  ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
> at 
> 

[jira] [Closed] (DRILL-6127) NullPointerException happens when submitting physical plan to the Hive storage plugin

[Anton Gozhiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Gozhiy closed DRILL-6127.
---

Hive fix tested with Drill version 1.13.0-SNAPSHOT (commit id: 
9073aed67d89e8b2188870d6c812706085c9c41b)
Cases verified:
 - Original issue wasn't reproduced.
 - Submitted logical plan
 - Tried with different queries (filters, agg functions, unions, joins etc)
 - Queries with 2+ major fragments worked fine

> NullPointerException happens when submitting physical plan to the Hive 
> storage plugin
> -
>
> Key: DRILL-6127
> URL: https://issues.apache.org/jira/browse/DRILL-6127
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.13.0
>
>
> *Prerequisites:*
> *1.* Create some test table in Hive:
> {code:sql}
> create external table if not exists hive_storage.test (key string, value 
> string) stored as parquet
> location '/hive_storage/test';
> insert into table test values ("key", "value");
> {code}
> *2.* Hive plugin config:
> {code:json}
> {
>   "type": "hive",
>   "enabled": true,
>   "configProps": {
> "hive.metastore.uris": "thrift://localhost:9083",
> "fs.default.name": "maprfs:///",
> "hive.metastore.sasl.enabled": "false"
>   }
> }
> {code}
> *Steps:*
> *1.* From the Drill web UI, run the following query:
> {code:sql}
> explain plan for select * from hive.hive_storage.`test`
> {code}
> *2.* Copy the json part of the plan
> *3.* On the Query page set checkbox to the PHYSICAL
> *4.* Submit the copied plan  
> *Expected result:*
> Drill should return normal result: "key", "value"
> *Actual result:*
> NPE happens:
> {noformat}
> [Error Id: 8b45c27e-bddd-4552-b7ea-e5af6f40866a on node1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> [Error Id: 8b45c27e-bddd-4552-b7ea-e5af6f40866a on node1:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:761)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:327)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:223)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:279) 
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: org.apache.drill.exec.work.foreman.ForemanSetupException: Failure 
> while parsing physical plan.
> at 
> org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan(Foreman.java:393)
>  [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257) 
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation 
> of [simple type, class org.apache.drill.exec.store.hive.HiveScan] value 
> failed (java.lang.NullPointerException): null
>  at [Source: { "head" : { "version" : 1, "generator" : { "type" : 
> "ExplainHandler", "info" : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" 
> : [ ], "queue" : 0, "hasResourcePlan" : false, "resultMode" : "EXEC" }, 
> "graph" : [ { "pop" : "hive-scan", "@id" : 2, "userName" : "mapr", 
> "hive-table" : { "table" : { "tableName" : "test", "dbName" : "hive_storage", 
> "owner" : "mapr", "createTime" : 1517417959, "lastAccessTime" : 0, 
> "retention" : 0, "sd" : { "location" : "maprfs:/hive_storage/test", 
> "inputFormat" : 
> "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", 
> "outputFormat" : 
> "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat", 
> "compressed" : false, "numBuckets" : -1, "serDeInfo" : { "name" : null, 
> "serializationLib" : 
> "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", "parameters" : 
> { "serialization.format" : "1" } }, "sortCols" : [ ], 

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400927#comment-16400927
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sachouche commented on a diff in the pull request:

https://github.com/apache/drill/pull/1168#discussion_r174889287
  
--- Diff: exec/jdbc-all/pom.xml ---
@@ -473,6 +473,8 @@
org/yaml/**
hello/**
webapps/**
+   **/org/apache/calcite/avatica/metrics/**
+   **/org/apache/calcite/avatica/org/**
--- End diff --

done!


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400889#comment-16400889
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1168#discussion_r174882554
  
--- Diff: exec/jdbc-all/pom.xml ---
@@ -473,6 +473,8 @@
org/yaml/**
hello/**
webapps/**
+   **/org/apache/calcite/avatica/metrics/**
+   **/org/apache/calcite/avatica/org/**
--- End diff --

Would be great to make that change. Other than that, changes looks good to 
me


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400812#comment-16400812
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sachouche commented on a diff in the pull request:

https://github.com/apache/drill/pull/1168#discussion_r174870229
  
--- Diff: exec/jdbc-all/pom.xml ---
@@ -473,6 +473,8 @@
org/yaml/**
hello/**
webapps/**
+   **/org/apache/calcite/avatica/metrics/**
+   **/org/apache/calcite/avatica/org/**
--- End diff --

The Apache jar size has shrunk by ~1MB from over 34MB to 33393541 bytes; I 
guess we can decrease the limit from 35MB to 34MB.


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6246) Build Failing in jdbc-all artifact

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-6246:
-
Reviewer: Sorabh Hamirwasia

> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400776#comment-16400776
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1168#discussion_r174866472
  
--- Diff: exec/jdbc-all/pom.xml ---
@@ -473,6 +473,8 @@
org/yaml/**
hello/**
webapps/**
+   **/org/apache/calcite/avatica/metrics/**
+   **/org/apache/calcite/avatica/org/**
--- End diff --

After removing these exclusion for apache profile what is the size of 
jdbc-all jar ? based on final size we should reduce the allowed maxSize from 
~35MB to that value.


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6251) Queries from system tables are hang

[Vitalii Diravka (JIRA)]

Vitalii Diravka created DRILL-6251:
--

 Summary: Queries from system tables are hang
 Key: DRILL-6251
 URL: https://issues.apache.org/jira/browse/DRILL-6251
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Vitalii Diravka


On CentoOS cluster Drill hangs while querying sys tables after "use dfs;" 
(embedded or distributed mode):
{code}
0: jdbc:drill:> select * from sys.version;
+--+---+--++++
| version  | commit_id |
commit_message|commit_time |build_email 
| build_time |
+--+---+--++++
| 1.13.0   | 796fcf051b3553c4597abbdca5ca247b139734ba  | [maven-release-plugin] 
prepare release drill-1.13.0  | 13.03.2018 @ 11:39:14 IST  | par...@apache.org  
| 13.03.2018 @ 13:13:45 IST  |
+--+---+--++++
1 row selected (3.784 seconds)
0: jdbc:drill:> use dfs;
+---+--+
|  ok   | summary  |
+---+--+
| true  | Default schema changed to [dfs]  |
+---+--+
1 row selected (0.328 seconds)
0: jdbc:drill:> select * from sys.version;
Error: Statement canceled (state=,code=0)
0: jdbc:drill:> 
{code}
*Note*: there is not failure on local debian machine in embedded mode.
dfs pugin configs are default (with "connection": "file:///", other file 
systems works good).

This failure is connected to DRILL-5089 and calcite rebase. 
Related commits:
https://github.com/apache/drill/commit/450e67094eb6e9a6484d7f86c49b51c77a08d7b2
https://github.com/apache/drill/commit/18a71a38f6bd1fd33d21d1c68fc23c5901b0080a

After analyzing in remote debug I found the following flow:
"dfs" DynamicRootSchema is created, then a new "sys" one is created.
After calcite validate "sys" SimpleCalciteSchema is created. But in 
WorkspaceSchemaFactory#create  wrong WorkspaceConfig is left and "/" is 
combined with "sys".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400750#comment-16400750
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

Github user sachouche commented on the issue:

https://github.com/apache/drill/pull/1168
  
@sohami can you please review this pull request?

Thanks!


> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6246) Build Failing in jdbc-all artifact

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400747#comment-16400747
 ] 

ASF GitHub Bot commented on DRILL-6246:
---

GitHub user sachouche opened a pull request:

https://github.com/apache/drill/pull/1168

DRILL-6246: Reduced the size of the jdbc-all jar file

- The jdbc-all client jar has been growing in size from version to the next 
(~20MB in version 1.10, ~27MB in version 1.12, and 34MB in version 1.13
- Note the exact size of the size depends on the maven profile used during 
compilation (e.g., "mapr" which have more excludes)
- Originally, I tried to exclude the following calcite/avatica packages 
(responsible for the recent size increase): metrics, proto, org/apache/, 
com/fasterxml, com/google but some tests failed as the calcite code was 
referencing some of them
- In this pull request, I have excluded the following packages: 
calcite/avatica/org and calcite/avatica/metrics

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sachouche/drill DRILL-6246

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1168


commit 6b6188a6c1677595c479f67fc2e33af091c36818
Author: Salim Achouche 
Date:   2018-03-14T02:16:06Z

DRILL-6246: Reduced the size of the jdbc-all jar file




> Build Failing in jdbc-all artifact
> --
>
> Key: DRILL-6246
> URL: https://issues.apache.org/jira/browse/DRILL-6246
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> * {color:#00}It was noticed that the build was failing because of the 
> jdbc-all artifact{color}
>  * {color:#00}The maximum compressed jar size was set to 32MB but we are 
> currently creating a JAR a bit larger than 32MB {color}
>  * {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and 
> drill-1.13.0 (on my MacOS){color}
>  * {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
>  * {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
>  * {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is 
> roughly 32MB){color}
>  * {color:#00}Compared then in more details jdbc-all-1.12.0 and 
> jdbc-all-1.13.0{color}
>  * {color:#00}The bulk of the increase is attributed to the calcite 
> artifact{color}
>  * {color:#00}Used to be 2MB (uncompressed) and now 22MB 
> (uncompressed){color}
>  * {color:#00}It is likely an exclusion problem {color}
>  * {color:#00}The jdbc-all-1.12.0 version has only two top packages 
> calcite/avatica/utils and calcite/avatica/remote{color}
>  * {color:#00}The jdbc-all-1.13.0  includes new packages (within 
> calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
> {color:#00} {color}
> {color:#00}I am planning to exclude these new sub-packages{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400151#comment-16400151
 ] 

ASF GitHub Bot commented on DRILL-6016:
---

Github user rajrahul commented on the issue:

https://github.com/apache/drill/pull/1166
  
@parthchandra @vdiravka I have added the test case using the same parquet 
file(2.9k bytes). I tried creating a smaller file using Spark, but could not 
replicate the behavior. I have rebased the changes on the same commit and PR.


> Error reading INT96 created by Apache Spark
> ---
>
> Key: DRILL-6016
> URL: https://issues.apache.org/jira/browse/DRILL-6016
> Project: Apache Drill
>  Issue Type: Bug
> Environment: Drill 1.11
>Reporter: Rahul Raj
>Priority: Major
> Fix For: 1.14.0
>
>
> Hi,
> I am getting the error - SYSTEM ERROR : ClassCastException: 
> org.apache.drill.exec.vector.TimeStampVector cannot be cast to 
> org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark 
> INT96 datetime field on Drill 1.11 in spite of setting the property 
> store.parquet.reader.int96_as_timestamp to  true.
> I believe this was fixed in drill 
> 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong.
> I have attached the dataset at 
> https://github.com/rajrahul/files/blob/master/result.tar.gz



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (DRILL-6189) Security: passwords logging and file permisions

[Anton Gozhiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Gozhiy closed DRILL-6189.
---

Verified with Drill version 1.13.0-SNAPSHOT (commit id: 
cac2882d5a9e22fbc251e4caf622fe30242ad557)

Cases verified:
 # Drillbit startup
 # Sqlline connection
 # JDBC connection to Drill
 # ODBC connection to Drill
 # Connection from Drill to MySQL

A new issues was reported: DRILL-6241, DRILL-6250

> Security: passwords logging and file permisions
> ---
>
> Key: DRILL-6189
> URL: https://issues.apache.org/jira/browse/DRILL-6189
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> *Prerequisites:*
>  *1.* Log level is set to "all" in the conf/logback.xml:
> {code:xml}
> 
> 
> 
> 
> {code}
> *2.* PLAIN authentication mechanism is configured:
> {code:java}
>   security.user.auth: {
>   enabled: true,
>   packages += "org.apache.drill.exec.rpc.user.security",
>   impl: "pam",
>   pam_profiles: [ "sudo", "login" ]
>   }
> {code}
> *Steps:*
>  *1.* Start the drillbits
>  *2.* Connect by sqlline:
> {noformat}
> /opt/mapr/drill/drill-1.13.0/bin/sqlline -u "jdbc:drill:zk=node1:5181;" -n 
> user1 -p 
> {noformat}
> *Expected result:* Logs shouldn't contain clear-text passwords
> *Actual results:* During the drillbit startup or establishing connections via 
> the jdbc or odbc, the following lines appear in the drillbit.log:
> {noformat}
> properties {
> key: "password"
> value: ""
> }
> {noformat}
> Same thing happens with storage configuration data, everything, including 
> passwords is being logged to file.
> *Another issue:*
> Currently Drill config files has the permissions 0644:
> {noformat}
> -rw-r--r--. 1 mapr mapr 1081 Nov 16 14:42 core-site-example.xml
> -rwxr-xr-x. 1 mapr mapr 1807 Dec 19 11:55 distrib-env.sh
> -rw-r--r--. 1 mapr mapr 1424 Nov 16 14:42 distrib-env.sh.prejmx
> -rw-r--r--. 1 mapr mapr 1942 Nov 16 14:42 drill-am-log.xml
> -rw-r--r--. 1 mapr mapr 1279 Dec 19 11:55 drill-distrib.conf
> -rw-r--r--. 1 mapr mapr  117 Nov 16 14:50 drill-distrib-mem-qs.conf
> -rw-r--r--. 1 mapr mapr 6016 Nov 16 14:42 drill-env.sh
> -rw-r--r--. 1 mapr mapr 1855 Nov 16 14:50 drill-on-yarn.conf
> -rw-r--r--. 1 mapr mapr 6913 Nov 16 14:42 drill-on-yarn-example.conf
> -rw-r--r--. 1 mapr mapr 1135 Dec 19 11:55 drill-override.conf
> -rw-r--r--. 1 mapr mapr 7820 Nov 16 14:42 drill-override-example.conf
> -rw-r--r--. 1 mapr mapr 3136 Nov 16 14:42 logback.xml
> -rw-r--r--. 1 mapr mapr  668 Nov 16 14:51 warden.drill-bits.conf
> -rw-r--r--. 1 mapr mapr 1581 Nov 16 14:42 yarn-client-log.xml
> {noformat}
> As they may contain some sensitive information, like passwords or secret 
> keys, they cannot be viewable to everyone. So I suggest to reduce the 
> permissions at least to 0640.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6250) Sqlline start command with password appears in the sqlline.log

[Anton Gozhiy (JIRA)]

Anton Gozhiy created DRILL-6250:
---

 Summary: Sqlline start command with password appears in the 
sqlline.log
 Key: DRILL-6250
 URL: https://issues.apache.org/jira/browse/DRILL-6250
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Anton Gozhiy


*Prerequisites:*
 *1.* Log level is set to "all" in the conf/logback.xml:
{code:xml}




{code}
*2.* PLAIN authentication mechanism is configured:
{code:java}
  security.user.auth: {
enabled: true,
packages += "org.apache.drill.exec.rpc.user.security",
impl: "pam",
pam_profiles: [ "sudo", "login" ]
  }
{code}
*Steps:*
 *1.* Start the drillbits
 *2.* Connect by sqlline:
{noformat}
/opt/mapr/drill/drill-1.13.0/bin/sqlline -u "jdbc:drill:zk=node1:5181;" -n 
user1 -p 1234
{noformat}
*3.* Use check the sqlline logs:
{noformat}
tail -F log/sqlline.log|grep 1234 -a5 -b5
{noformat}
*Expected result:* Logs shouldn't contain clear-text passwords

*Actual result:* The logs contain the sqlline start command with password:
{noformat}
# system properties
35333-"java" : {
35352-# system properties
35384:"command" : "sqlline.SqlLine -d org.apache.drill.jdbc.Driver 
--maxWidth=1 --color=true -u jdbc:drill:zk=node1:5181; -n user1 -p 1234",
35535-# system properties
35567-"launcher" : "SUN_STANDARD"
35607-}
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)