[jira] [Created] (DRILL-4336) Fix weird interaction between maven-release, maven-enforcer and RAT plugins

[Jason Altekruse (JIRA)]

Jason Altekruse created DRILL-4336:
--

 Summary: Fix weird interaction between maven-release, 
maven-enforcer and RAT plugins
 Key: DRILL-4336
 URL: https://issues.apache.org/jira/browse/DRILL-4336
 Project: Apache Drill
  Issue Type: Bug
Reporter: Jason Altekruse


While trying to make the 1.5.0 release I ran into a bizarre failure from RAT 
complaining about a file it should have been ignoring according to the plugin 
configuration.

Disabling the newly added maven-enforcer plugin "fixed" the issue, but we need 
to keep this in the build to make sure new dependencies don't creep into the 
JDBC driver that is supposed to be as small as possible.

For the sake of the release the jdbc-all jar's size was checked manually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4326) JDBC Storage Plugin for PostgreSQL does not work

[Akon Dey (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127045#comment-15127045
 ] 

Akon Dey commented on DRILL-4326:
-

On further investigation, I found that this actually works if you use the table 
name {{public.ips}} instead of {{ips}}. This should be documented somewhere. It 
will help newbies start using Drill more easily.

There is still a usability issue which should be fixed or appropriately 
documented. The {{show tables;}} command lists the table with name {{its}}, 
however, to access it one must qualify the table name with its schema name, in 
this case {{public}} using {{public.ps}}.

Please see the log below for details.
 
{noformat}
0: jdbc:drill:zk=local> use pgdb;
use pgdb;
1 row selected (0.461 seconds)
+---+---+
|  ok   |  summary  |
+---+---+
| true  | Default schema changed to [pgdb]  |
+---+---+
0: jdbc:drill:zk=local> show tables;
show tables;
59 rows selected (0.687 seconds)
+---+--+
| TABLE_SCHEMA  |TABLE_NAME|
+---+--+
| pgdb.test | ips  |
| pgdb.test | pg_aggregate |
| pgdb.test | pg_am|
| pgdb.test | pg_amop  |
| pgdb.test | pg_amproc|
| pgdb.test | pg_attrdef   |
| pgdb.test | pg_attribute |
| pgdb.test | pg_auth_members  |
| pgdb.test | pg_authid|
| pgdb.test | pg_cast  |
| pgdb.test | pg_class |
| pgdb.test | pg_collation |
| pgdb.test | pg_constraint|
| pgdb.test | pg_conversion|
| pgdb.test | pg_database  |
| pgdb.test | pg_db_role_setting   |
| pgdb.test | pg_default_acl   |
| pgdb.test | pg_depend|
| pgdb.test | pg_description   |
| pgdb.test | pg_enum  |
| pgdb.test | pg_event_trigger |
| pgdb.test | pg_extension |
| pgdb.test | pg_foreign_data_wrapper  |
| pgdb.test | pg_foreign_server|
| pgdb.test | pg_foreign_table |
| pgdb.test | pg_index |
| pgdb.test | pg_inherits  |
| pgdb.test | pg_language  |
| pgdb.test | pg_largeobject   |
| pgdb.test | pg_largeobject_metadata  |
| pgdb.test | pg_namespace |
| pgdb.test | pg_opclass   |
| pgdb.test | pg_operator  |
| pgdb.test | pg_opfamily  |
| pgdb.test | pg_pltemplate|
| pgdb.test | pg_proc  |
| pgdb.test | pg_range |
| pgdb.test | pg_rewrite   |
| pgdb.test | pg_seclabel  |
| pgdb.test | pg_shdepend  |
| pgdb.test | pg_shdescription |
| pgdb.test | pg_shseclabel|
| pgdb.test | pg_statistic |
| pgdb.test | pg_tablespace|
| pgdb.test | pg_trigger   |
| pgdb.test | pg_ts_config |
| pgdb.test | pg_ts_config_map |
| pgdb.test | pg_ts_dict   |
| pgdb.test | pg_ts_parser |
| pgdb.test | pg_ts_template   |
| pgdb.test | pg_type  |
| pgdb.test | pg_user_mapping  |
| pgdb.test | sql_features |
| pgdb.test | sql_implementation_info  |
| pgdb.test | sql_languages|
| pgdb.test | sql_packages |
| pgdb.test | sql_parts|
| pgdb.test | sql_sizing   |
| pgdb.test | sql_sizing_profiles  |
+---+--+
0: jdbc:drill:zk=local> select * from pgdb.test.ips;
select * from pgdb.test.ips;
0: jdbc:drill:zk=local> Error: DATA_READ ERROR: The JDBC storage plugin failed 
while trying setup the SQL query. 

sql SELECT *
FROM "test"."ips"
plugin pgdb
Fragment 0:0

[Error Id: 5455fd05-6b25-4489-bc5c-5615db197b9f on 10.200.104.128:31010] 
(state=,code=0)


0: jdbc:drill:zk=local> select * from public.ips;
select * from public.ips;
2 rows selected (0.121 seconds)
+---+--+
| ipid  | ipv4dot  |
+---+--+
| 1 | 1.2.3.4  |
| 2 | 1.2.3.5  |
+---+--+
0: jdbc:drill:zk=local> 
{noformat}

> JDBC Storage Plugin for PostgreSQL does not work
> 
>
> Key: DRILL-4326
> URL: https://issues.apache.org/jira/browse/DRILL-4326
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
> Environment: Mac OS X JDK 1.8 

[jira] [Commented] (DRILL-4069) Enable RPC Thread Offload by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127041#comment-15127041
 ] 

ASF GitHub Bot commented on DRILL-4069:
---

GitHub user sudheeshkatkam opened a pull request:

https://github.com/apache/drill/pull/352

DRILL-4069: Enable RPC thread offload by default


https://github.com/apache/drill/commit/43414e1e2731c85a52febaaedebfa2f392dd6d2f 
removes the usage of recyclers for RequestEvent and ResponseEvent in the RPC 
layer. This change resolves functional regressions mentioned in 
[DRILL-4041](https://issues.apache.org/jira/browse/DRILL-4041) and 
[DRILL-4057](https://issues.apache.org/jira/browse/DRILL-4057).

Multiple test runs with the same setup mentioned in the tickets did not 
show any regressions.

@jacques-n can you please review?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sudheeshkatkam/drill DRILL-4069

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #352


commit f2d42074d64d2146eb15bee2374edf78864f832e
Author: Sudheesh Katkam 
Date:   2016-01-22T21:12:50Z

DRILL-4069: Enable RPC thread offload by default




> Enable RPC Thread Offload by default
> 
>
> Key: DRILL-4069
> URL: https://issues.apache.org/jira/browse/DRILL-4069
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Reporter: Jacques Nadeau
>
> Once we enabled RPC thread offload, we saw concurrency regressions DRILL-4041 
> and DRILL-4057. The regressions appear to be unrelated to, but exposed by the 
> thread offload. To simplify things, we disabled the RPC thread offload by 
> default. This is the tracking issue to fix the underlying concurrency bug(s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4287) Do lazy reading of parquet metadata cache file

[Aman Sinha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aman Sinha updated DRILL-4287:
--
Assignee: Jinfeng Ni  (was: Aman Sinha)

> Do lazy reading of parquet metadata cache file
> --
>
> Key: DRILL-4287
> URL: https://issues.apache.org/jira/browse/DRILL-4287
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4069) Enable RPC Thread Offload by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127063#comment-15127063
 ] 

ASF GitHub Bot commented on DRILL-4069:
---

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/352#issuecomment-178199199
  
LGTM. +1

Nice (but strange) that this resolves the other issues.

I wonder if the local channel short circuit would work now...


> Enable RPC Thread Offload by default
> 
>
> Key: DRILL-4069
> URL: https://issues.apache.org/jira/browse/DRILL-4069
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Reporter: Jacques Nadeau
>
> Once we enabled RPC thread offload, we saw concurrency regressions DRILL-4041 
> and DRILL-4057. The regressions appear to be unrelated to, but exposed by the 
> thread offload. To simplify things, we disabled the RPC thread offload by 
> default. This is the tracking issue to fix the underlying concurrency bug(s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4335) Apache Drill should support network encryption

[Keys Botzum (JIRA)]

Keys Botzum created DRILL-4335:
--

 Summary: Apache Drill should support network encryption
 Key: DRILL-4335
 URL: https://issues.apache.org/jira/browse/DRILL-4335
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Keys Botzum


This is clearly related to Drill-291 but wanted to make explicit that this 
needs to include network level encryption and not just authentication. This is 
particularly important for the client connection to Drill which will often be 
sending passwords in the clear until there is encryption.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4337) Drill fails to read INT96 fields from hive generated parquet files

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127212#comment-15127212
 ] 

Rahul Challapalli commented on DRILL-4337:
--

Failure 2 :
{code}
2016-02-01 21:38:58,089 [29502f8d-bb52-bd78-51a4-bdbf14f3a498:frag:0:0] DEBUG 
o.a.d.exec.physical.impl.ScanBatch - Failed to read the batch. Stopping...
org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet 
record reader.
Message:
Hadoop path: 
/drill/testdata/hive_storage/hive1_fewtypes_null/hive1_fewtypes_null.parquet
Total records read: 0
Mock records read: 0
Records to read: 21
Row group index: 0
Records in row group: 21
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
  optional int32 int_col;
  optional int64 bigint_col;
  optional binary date_col (UTF8);
  optional binary time_col (UTF8);
  optional int96 timestamp_col;
  optional binary interval_col (UTF8);
  optional binary varchar_col (UTF8);
  optional float float_col;
  optional double double_col;
  optional boolean bool_col;
}
, metadata: {}}, blocks: [BlockMetaData{21, 1886 [ColumnMetaData{UNCOMPRESSED 
[int_col] INT32  [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData{UNCOMPRESSED 
[bigint_col] INT64  [RLE, BIT_PACKED, PLAIN], 111}, ColumnMetaData{UNCOMPRESSED 
[date_col] BINARY  [RLE, BIT_PACKED, PLAIN], 298}, ColumnMetaData{UNCOMPRESSED 
[time_col] BINARY  [RLE, BIT_PACKED, PLAIN_DICTIONARY], 563}, 
ColumnMetaData{UNCOMPRESSED [timestamp_col] INT96  [RLE, BIT_PACKED, 
PLAIN_DICTIONARY], 793}, ColumnMetaData{UNCOMPRESSED [interval_col] BINARY  
[RLE, BIT_PACKED, PLAIN_DICTIONARY], 1031}, ColumnMetaData{UNCOMPRESSED 
[varchar_col] BINARY  [RLE, BIT_PACKED, PLAIN], 1189}, 
ColumnMetaData{UNCOMPRESSED [float_col] FLOAT  [RLE, BIT_PACKED, PLAIN], 1543}, 
ColumnMetaData{UNCOMPRESSED [double_col] DOUBLE  [RLE, BIT_PACKED, PLAIN], 
1654}, ColumnMetaData{UNCOMPRESSED [bool_col] BOOLEAN  [RLE, BIT_PACKED, 
PLAIN], 1851}]}]}
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleAndRaise(ParquetRecordReader.java:349)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:451)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:191) 
~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.7.0_71]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1506.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_71]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Reading past RLE/BitPacking 
stream.
at 

[jira] [Commented] (DRILL-4337) Drill fails to read INT96 fields from hive generated parquet files

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127215#comment-15127215
 ] 

Rahul Challapalli commented on DRILL-4337:
--

Failure 3 :
{code}
org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet 
record reader.
Message:
Hadoop path: 
/drill/testdata/hive_storage/hive1_fewtypes_null/hive1_fewtypes_null.parquet
Total records read: 0
Mock records read: 0
Records to read: 21
Row group index: 0
Records in row group: 21
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
  optional int32 int_col;
  optional int64 bigint_col;
  optional binary date_col (UTF8);
  optional binary time_col (UTF8);
  optional int96 timestamp_col;
  optional binary interval_col (UTF8);
  optional binary varchar_col (UTF8);
  optional float float_col;
  optional double double_col;
  optional boolean bool_col;
}
, metadata: {}}, blocks: [BlockMetaData{21, 1886 [ColumnMetaData{UNCOMPRESSED 
[int_col] INT32  [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData{UNCOMPRESSED 
[bigint_col] INT64  [RLE, BIT_PACKED, PLAIN], 111}, ColumnMetaData{UNCOMPRESSED 
[date_col] BINARY  [RLE, BIT_PACKED, PLAIN], 298}, ColumnMetaData{UNCOMPRESSED 
[time_col] BINARY  [RLE, BIT_PACKED, PLAIN_DICTIONARY], 563}, 
ColumnMetaData{UNCOMPRESSED [timestamp_col] INT96  [RLE, BIT_PACKED, 
PLAIN_DICTIONARY], 793}, ColumnMetaData{UNCOMPRESSED [interval_col] BINARY  
[RLE, BIT_PACKED, PLAIN_DICTIONARY], 1031}, ColumnMetaData{UNCOMPRESSED 
[varchar_col] BINARY  [RLE, BIT_PACKED, PLAIN], 1189}, 
ColumnMetaData{UNCOMPRESSED [float_col] FLOAT  [RLE, BIT_PACKED, PLAIN], 1543}, 
ColumnMetaData{UNCOMPRESSED [double_col] DOUBLE  [RLE, BIT_PACKED, PLAIN], 
1654}, ColumnMetaData{UNCOMPRESSED [bool_col] BOOLEAN  [RLE, BIT_PACKED, 
PLAIN], 1851}]}]}
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleAndRaise(ParquetRecordReader.java:349)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:451)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:191) 
~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.7.0_71]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1506.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_71]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at 
org.apache.parquet.column.values.rle.RunLengthBitPackingHybridDecoder.readInt(RunLengthBitPackingHybridDecoder.java:75)
 ~[parquet-column-1.8.1-drill-r0.jar:1.8.1-drill-r0]
at 

[jira] [Created] (DRILL-4337) Drill fails to read INT96 fields from hive generated parquet files

[Rahul Challapalli (JIRA)]

Rahul Challapalli created DRILL-4337:


 Summary: Drill fails to read INT96 fields from hive generated 
parquet files
 Key: DRILL-4337
 URL: https://issues.apache.org/jira/browse/DRILL-4337
 Project: Apache Drill
  Issue Type: Bug
Reporter: Rahul Challapalli
Priority: Critical


git.commit.id.abbrev=576271d
Cluster : 2 nodes running MaprFS 4.1

The data file used in the below table is generated from hive. Below is output 
from running the same query multiple times. 
{code}
0: jdbc:drill:zk=10.10.100.190:5181> select timestamp_col from 
hive1_fewtypes_null;
Error: SYSTEM ERROR: NegativeArraySizeException

Fragment 0:0

[Error Id: 5517e983-ccae-4c96-b09c-30f331919e56 on qa-node191.qa.lab:31010] 
(state=,code=0)

0: jdbc:drill:zk=10.10.100.190:5181> select timestamp_col from 
hive1_fewtypes_null;
Error: SYSTEM ERROR: IllegalArgumentException: Reading past RLE/BitPacking 
stream.

Fragment 0:0

[Error Id: 94ed5996-d2ac-438d-b460-c2d2e41bdcc3 on qa-node191.qa.lab:31010] 
(state=,code=0)

0: jdbc:drill:zk=10.10.100.190:5181> select timestamp_col from 
hive1_fewtypes_null;
Error: SYSTEM ERROR: ArrayIndexOutOfBoundsException: 0

Fragment 0:0

[Error Id: 41dca093-571e-49e5-a2ab-fd69210b143d on qa-node191.qa.lab:31010] 
(state=,code=0)

0: jdbc:drill:zk=10.10.100.190:5181> select timestamp_col from 
hive1_fewtypes_null;
++
| timestamp_col  |
++
| null   |
| [B@7c766115|
| [B@3fdfe989|
| null   |
| [B@55d4222 |
| [B@2da0c8ee|
| [B@16e798a9|
| [B@3ed78afe|
| [B@38e649ed|
| [B@16ff83ca|
| [B@61254e91|
| [B@5849436a|
| [B@31e9116e|
| [B@3c77665b|
| [B@42e0ff60|
| [B@419e19ed|
| [B@72b83842|
| [B@1c75afe5|
| [B@726ef1fb|
| [B@51d0d06e|
| [B@64240fb8|
+
{code}

Attached the log, hive ddl used to generate the parquet file and the parquet 
file itself



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DRILL-4337) Drill fails to read INT96 fields from hive generated parquet files

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127207#comment-15127207
 ] 

Rahul Challapalli edited comment on DRILL-4337 at 2/1/16 11:03 PM:
---

Failure1 :
{code}
org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet 
record reader.
Message:
Hadoop path: 
/drill/testdata/hive_storage/hive1_fewtypes_null/hive1_fewtypes_null.parquet
Total records read: 0
Mock records read: 0
Records to read: 21
Row group index: 0
Records in row group: 21
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
  optional int32 int_col;
  optional int64 bigint_col;
  optional binary date_col (UTF8);
  optional binary time_col (UTF8);
  optional int96 timestamp_col;
  optional binary interval_col (UTF8);
  optional binary varchar_col (UTF8);
  optional float float_col;
  optional double double_col;
  optional boolean bool_col;
}
, metadata: {}}, blocks: [BlockMetaData{21, 1886 [ColumnMetaData{UNCOMPRESSED 
[int_col] INT32  [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData{UNCOMPRESSED 
[bigint_col] INT64  [RLE, BIT_PACKED, PLAIN], 111}, ColumnMetaData{UNCOMPRESSED 
[date_col] BINARY  [RLE, BIT_PACKED, PLAIN], 298}, ColumnMetaData{UNCOMPRESSED 
[time_col] BINARY  [RLE, BIT_PACKED, PLAIN_DICTIONARY], 563}, 
ColumnMetaData{UNCOMPRESSED [timestamp_col] INT96  [RLE, BIT_PACKED, 
PLAIN_DICTIONARY], 793}, ColumnMetaData{UNCOMPRESSED [interval_col] BINARY  
[RLE, BIT_PACKED, PLAIN_DICTIONARY], 1031}, ColumnMetaData{UNCOMPRESSED 
[varchar_col] BINARY  [RLE, BIT_PACKED, PLAIN], 1189}, 
ColumnMetaData{UNCOMPRESSED [float_col] FLOAT  [RLE, BIT_PACKED, PLAIN], 1543}, 
ColumnMetaData{UNCOMPRESSED [double_col] DOUBLE  [RLE, BIT_PACKED, PLAIN], 
1654}, ColumnMetaData{UNCOMPRESSED [bool_col] BOOLEAN  [RLE, BIT_PACKED, 
PLAIN], 1851}]}]}
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleAndRaise(ParquetRecordReader.java:349)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:451)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:191) 
~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.7.0_71]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1506.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_71]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.NegativeArraySizeException: null
at 
org.apache.parquet.column.values.rle.RunLengthBitPackingHybridDecoder.readNext(RunLengthBitPackingHybridDecoder.java:97)
 

[jira] [Updated] (DRILL-4337) Drill fails to read INT96 fields from hive generated parquet files

[Rahul Challapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-4337:
-
Attachment: hive1_fewtypes_null.parquet

Failure1 :
{code}
org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet 
record reader.
Message:
Hadoop path: 
/drill/testdata/hive_storage/hive1_fewtypes_null/hive1_fewtypes_null.parquet
Total records read: 0
Mock records read: 0
Records to read: 21
Row group index: 0
Records in row group: 21
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message hive_schema {
  optional int32 int_col;
  optional int64 bigint_col;
  optional binary date_col (UTF8);
  optional binary time_col (UTF8);
  optional int96 timestamp_col;
  optional binary interval_col (UTF8);
  optional binary varchar_col (UTF8);
  optional float float_col;
  optional double double_col;
  optional boolean bool_col;
}
, metadata: {}}, blocks: [BlockMetaData{21, 1886 [ColumnMetaData{UNCOMPRESSED 
[int_col] INT32  [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData{UNCOMPRESSED 
[bigint_col] INT64  [RLE, BIT_PACKED, PLAIN], 111}, ColumnMetaData{UNCOMPRESSED 
[date_col] BINARY  [RLE, BIT_PACKED, PLAIN], 298}, ColumnMetaData{UNCOMPRESSED 
[time_col] BINARY  [RLE, BIT_PACKED, PLAIN_DICTIONARY], 563}, 
ColumnMetaData{UNCOMPRESSED [timestamp_col] INT96  [RLE, BIT_PACKED, 
PLAIN_DICTIONARY], 793}, ColumnMetaData{UNCOMPRESSED [interval_col] BINARY  
[RLE, BIT_PACKED, PLAIN_DICTIONARY], 1031}, ColumnMetaData{UNCOMPRESSED 
[varchar_col] BINARY  [RLE, BIT_PACKED, PLAIN], 1189}, 
ColumnMetaData{UNCOMPRESSED [float_col] FLOAT  [RLE, BIT_PACKED, PLAIN], 1543}, 
ColumnMetaData{UNCOMPRESSED [double_col] DOUBLE  [RLE, BIT_PACKED, PLAIN], 
1654}, ColumnMetaData{UNCOMPRESSED [bool_col] BOOLEAN  [RLE, BIT_PACKED, 
PLAIN], 1851}]}]}
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleAndRaise(ParquetRecordReader.java:349)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:451)
 ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:191) 
~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
[na:1.7.0_71]
at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
 [hadoop-common-2.7.0-mapr-1506.jar:na]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
 [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
[na:1.7.0_71]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
{code}

> Drill fails to read INT96 fields from hive generated parquet files
> --
>
> Key: DRILL-4337
> URL: 

[jira] [Closed] (DRILL-4278) Memory leak when using LIMIT

[Victoria Markman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victoria Markman closed DRILL-4278.
---

> Memory leak when using LIMIT
> 
>
> Key: DRILL-4278
> URL: https://issues.apache.org/jira/browse/DRILL-4278
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.4.0, 1.5.0
> Environment: OS X
> 0: jdbc:drill:zk=local> select * from sys.version;
> +--+---+-++++
> | version  | commit_id |   
> commit_message|commit_time |
> build_email | build_time |
> +--+---+-++++
> | 1.4.0| 32b871b24c7b69f59a1d2e70f444eed6e599e825  | 
> [maven-release-plugin] prepare release drill-1.4.0  | 08.12.2015 @ 00:24:59 
> PST  | venki.koruka...@gmail.com  | 08.12.2015 @ 01:14:39 PST  |
> +--+---+-++++
> 0: jdbc:drill:zk=local> select * from sys.options where status <> 'DEFAULT';
> +-+---+-+--+--+-+---++
> |name | kind  |  type   |  status  | num_val  | 
> string_val  | bool_val  | float_val  |
> +-+---+-+--+--+-+---++
> | planner.slice_target| LONG  | SYSTEM  | CHANGED  | 10   | null  
>   | null  | null   |
> | planner.width.max_per_node  | LONG  | SYSTEM  | CHANGED  | 5| null  
>   | null  | null   |
> +-+---+-+--+--+-+---++
> 2 rows selected (0.16 seconds)
>Reporter: jean-claude
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>
> copy the parquet files in the samples directory so that you have a 12 or so
> $ ls -lha /apache-drill-1.4.0/sample-data/nationsMF/
> nationsMF1.parquet
> nationsMF2.parquet
> nationsMF3.parquet
> create a file with a few thousand lines like these
> select * from dfs.`/Users/jccote/apache-drill-1.4.0/sample-data/nationsMF` 
> limit 500;
> start drill
> $ /apache-drill-1.4.0/bin/drill-embeded
> reduce the slice target size to force drill to use multiple fragment/threads
> jdbc:drill:zk=local> system set planner.slice_target=10;
> now run the list of queries from the file your created above
> jdbc:drill:zk=local> !run /Users/jccote/test-memory-leak-using-limit.sql
> the java heap space keeps going up until the old space is at 100% and 
> eventually you get an OutOfMemoryException in drill
> $ jstat -gccause 86850 5s
>   S0 S1 E  O  M CCSYGC YGCTFGCFGCT 
> GCTLGCC GCC 
>   0.00   0.00 100.00 100.00  98.56  96.71   2279   26.682   240  458.139  
> 484.821 GCLocker Initiated GC Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   242  461.347  
> 488.028 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   245  466.630  
> 493.311 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   247  470.020  
> 496.702 Allocation Failure   Ergonomics  
> If you do the same test but do not use the LIMIT then the memory usage does 
> not go up.
> If you add a where clause so that no results are returned, then the memory 
> usage does not go up.
> Something with the RPC layer?
> Also it seems sensitive to the number of fragments/threads. If you limit it 
> to one fragment/thread the memory usage goes up much slower.
> I have used parquet files and CSV files. In either case the behaviour is the 
> same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3353) Non data-type related schema changes errors

[Chun Chang (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127389#comment-15127389
 ] 

Chun Chang commented on DRILL-3353:
---

1.5.0-SNAPSHOT  | 9ff947288f3214fe8e525e001d89a4f91b8b0728

{noformat}
0: jdbc:drill:schema=dfs.drillTestDir> select log.event.attributes from 
dfs.`/drill/testdata/drill-3353/jira-3353.json` as log where log.si = 
'07A3F985-4B34-4A01-9B83-3B14548EF7BE' and log.type ='Opens App';
++
| EXPR$0 |
++
| 
{"logged":"no","wearable":"no","type":"organic","daysSinceInstall":"0","destination":"none","nth":"1"}
 |
| 
{"logged":"no","wearable":"no","type":"facebook","daysSinceInstall":"0","destination":"none","nth":"2","ad":"Teste-FB-Engagement-Puro-iOS-230615","adSet":"Teste-FB-Engagement-Puro-iOS-230615","campaign":"Teste-FB-Engagement-Puro-iOS-230615","channel":"Facebook-App-Engagement"}
 |
| 
{"logged":"no","wearable":"no","type":"facebook","daysSinceInstall":"0","destination":"none","nth":"3"}
 |
| 
{"logged":"no","wearable":"no","type":"branch","daysSinceInstall":"0","destination":"none","nth":"4","adSet":"Teste-Adwords-Engagement-Branch-iOS-230615-adset","campaign":"Teste-Adwords-Engagement-Branch-iOS-230615","channel":"Adwords"}
 |
| 
{"logged":"no","wearable":"no","type":"organic","daysSinceInstall":"0","destination":"none","nth":"5"}
 |
++
{noformat}

> Non data-type related schema changes errors
> ---
>
> Key: DRILL-3353
> URL: https://issues.apache.org/jira/browse/DRILL-3353
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - JSON
>Affects Versions: 1.0.0
>Reporter: Oscar Bernal
>Assignee: Steven Phillips
> Fix For: 1.5.0
>
> Attachments: i-bfbc0a5c-ios-PulsarEvent-2015-06-23_19.json.zip
>
>
> I'm having trouble querying a data set with varying schema for a nested 
> object fields. The majority of my data for a specific type of record has the 
> following nested data:
> {code}
> "attributes":{"daysSinceInstall":0,"destination":"none","logged":"no","nth":1,"type":"organic","wearable":"no"}}
> {code}
> Among those records (hundreds of them) I have only two with a slightly 
> different schema:
> {code}
> "attributes":{"adSet":"Teste-Adwords-Engagement-Branch-iOS-230615-adset","campaign":"Teste-Adwords-Engagement-Branch-iOS-230615","channel":"Adwords","daysSinceInstall":0,"destination":"none","logged":"no","nth":4,"type":"branch","wearable":"no"}}
> {code}
> When trying to query the "new" fields, my queries fail:
> With {code:sql}ALTER SYSTEM SET `store.json.all_text_mode` = true;{code}
> {noformat}
> 0: jdbc:drill:zk=local> select log.event.attributes from 
> `dfs`.`root`.`/file.json` as log where log.si = 
> '07A3F985-4B34-4A01-9B83-3B14548EF7BE' and log.event.attributes.ad = 
> 'Teste-FB-Engagement-Puro-iOS-230615';
> Error: SYSTEM ERROR: java.lang.NumberFormatException: 
> Teste-FB-Engagement-Puro-iOS-230615"
> Fragment 0:0
> [Error Id: 22d37a65-7dd0-4661-bbfc-7a50bbee9388 on 
> ip-10-0-1-16.sa-east-1.compute.internal:31010] (state=,code=0)
> {noformat}
> With {code:sql}ALTER SYSTEM SET `store.json.all_text_mode` = false;`{code}
> {noformat}
> 0: jdbc:drill:zk=local> select log.event.attributes from 
> `dfs`.`root`.`/file.json` as log where log.si = 
> '07A3F985-4B34-4A01-9B83-3B14548EF7BE';
> Error: DATA_READ ERROR: Error parsing JSON - You tried to write a Bit type 
> when you are using a ValueWriter of type NullableVarCharWriterImpl.
> File  file.json
> Record  35
> Fragment 0:0
> [Error Id: 5746e3e9-48c0-44b1-8e5f-7c94e7c64d0f on 
> ip-10-0-1-16.sa-east-1.compute.internal:31010] (state=,code=0)
> {noformat}
> If I try to extract all "attributes" from those events, Drill will only 
> return a subset of the fields, ignoring the others. 
> {noformat}
> 0: jdbc:drill:zk=local> select log.event.attributes from 
> `dfs`.`root`.`/file.json` as log where log.si = 
> '07A3F985-4B34-4A01-9B83-3B14548EF7BE' and log.type ='Opens App';
> ++
> |   EXPR$0   |
> ++
> | {"logged":"no","wearable":"no","type":""}   |
> | {"logged":"no","wearable":"no","type":""}  |
> | {"logged":"no","wearable":"no","type":""}  |
> | {"logged":"no","wearable":"no","type":""}|
> | {"logged":"no","wearable":"no","type":""}   |
> ++
> {noformat}
> What I find strange is that I have thousands of records in the same file with 
> different schema for different record types and all other queries seem run 
> well.
> Is there something about how Drill infers schema that I might be missing 
> here? Does it infer based on a sample % of the data and fail for records that 
> were not taken into account while inferring 

[jira] [Commented] (DRILL-4128) null pointer at org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127496#comment-15127496
 ] 

ASF GitHub Bot commented on DRILL-4128:
---

GitHub user jaltekruse opened a pull request:

https://github.com/apache/drill/pull/353

DRILL-4128: Fix NPE when calling getString on a JDBC ResultSet when t…

…he type is not varchar

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaltekruse/incubator-drill 4128-jdbc-npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #353


commit 1b96174b1e5bafb13a873dd79f03467802d7c929
Author: Jason Altekruse 
Date:   2016-02-01T21:03:23Z

DRILL-4128: Fix NPE when calling getString on a JDBC ResultSet when the 
type is not varchar




> null pointer at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> -
>
> Key: DRILL-4128
> URL: https://issues.apache.org/jira/browse/DRILL-4128
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
>Reporter: Devender Yadav 
>Priority: Blocker
> Fix For: Future
>
>
> While fetching data from ResultSet in JDBC. I got the null pointer. Details - 
> java.lang.NullPointerException
> at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> at 
> org.apache.drill.exec.vector.accessor.BoundCheckingAccessor.getString(BoundCheckingAccessor.java:124)
> at 
> org.apache.drill.jdbc.impl.TypeConvertingSqlAccessor.getString(TypeConvertingSqlAccessor.java:649)
> at 
> org.apache.drill.jdbc.impl.AvaticaDrillSqlAccessor.getString(AvaticaDrillSqlAccessor.java:95)
> at 
> net.hydromatic.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:205)
> at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.getString(DrillResultSetImpl.java:182)
> Below mentioned method is throwing null pointer becaue getObject(rowOffset) 
> returns null for null values & null.toString() is throwing null pointer.
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset).toString();
>   }
> It should be like:
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset)==null? null:getObject(rowOffset).toString();
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4128) null pointer at org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127568#comment-15127568
 ] 

ASF GitHub Bot commented on DRILL-4128:
---

Github user jacques-n commented on the pull request:

https://github.com/apache/drill/pull/353#issuecomment-178337300
  
+1. yay!


> null pointer at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> -
>
> Key: DRILL-4128
> URL: https://issues.apache.org/jira/browse/DRILL-4128
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
>Reporter: Devender Yadav 
>Priority: Blocker
> Fix For: Future
>
>
> While fetching data from ResultSet in JDBC. I got the null pointer. Details - 
> java.lang.NullPointerException
> at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> at 
> org.apache.drill.exec.vector.accessor.BoundCheckingAccessor.getString(BoundCheckingAccessor.java:124)
> at 
> org.apache.drill.jdbc.impl.TypeConvertingSqlAccessor.getString(TypeConvertingSqlAccessor.java:649)
> at 
> org.apache.drill.jdbc.impl.AvaticaDrillSqlAccessor.getString(AvaticaDrillSqlAccessor.java:95)
> at 
> net.hydromatic.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:205)
> at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.getString(DrillResultSetImpl.java:182)
> Below mentioned method is throwing null pointer becaue getObject(rowOffset) 
> returns null for null values & null.toString() is throwing null pointer.
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset).toString();
>   }
> It should be like:
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset)==null? null:getObject(rowOffset).toString();
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4132) Ability to submit simple type of physical plan directly to EndPoint DrillBit for execution

[Yuliya Feldman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Feldman updated DRILL-4132:
--
Component/s: Query Planning & Optimization

> Ability to submit simple type of physical plan directly to EndPoint DrillBit 
> for execution
> --
>
> Key: DRILL-4132
> URL: https://issues.apache.org/jira/browse/DRILL-4132
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow, Execution - RPC, Query Planning & 
> Optimization
>Reporter: Yuliya Feldman
>Assignee: Yuliya Feldman
>
> Today Drill Query execution is optimistic and stateful (at least due to data 
> exchanges) - if any of the stages of query execution fails whole query fails. 
> If query is just simple scan, filter push down and project where no data 
> exchange happens between DrillBits there is no need to fail whole query when 
> one DrillBit fails, as minor fragments running on that DrillBit can be rerun 
> on the other DrillBit. There are probably multiple ways to achieve this. This 
> JIRA is to open discussion on: 
> 1. agreement that we need to support above use case 
> 2. means of achieving it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4278) Memory leak when using LIMIT

[Victoria Markman (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127355#comment-15127355
 ] 

Victoria Markman commented on DRILL-4278:
-

Verified fixed in: 1.5.0

{code}
#Generated by Git-Commit-Id-Plugin
#Sat Jan 30 04:56:47 UTC 2016
git.commit.id.abbrev=9ff9472
git.commit.user.email=j...@apache.org
git.commit.message.full=DRILL-4279\: Improve performance for skipAll query 
against Text/JSON/Parquet table.\n
git.commit.id=9ff947288f3214fe8e525e001d89a4f91b8b0728
git.commit.message.short=DRILL-4279\: Improve performance for skipAll query 
against Text/JSON/Parquet table.
git.commit.user.name=Jinfeng Ni
git.build.user.name=Unknown
git.commit.id.describe=0.9.0-581-g9ff9472-dirty
git.build.user.email=Unknown
git.branch=master
git.commit.time=30.01.2016 @ 03\:30\:27 UTC
git.build.time=30.01.2016 @ 04\:56\:47 UTC
git.remote.origin.url=https\://github.com/apache/drill
{code}

Will automated with other tests for men leak detection.

> Memory leak when using LIMIT
> 
>
> Key: DRILL-4278
> URL: https://issues.apache.org/jira/browse/DRILL-4278
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.4.0, 1.5.0
> Environment: OS X
> 0: jdbc:drill:zk=local> select * from sys.version;
> +--+---+-++++
> | version  | commit_id |   
> commit_message|commit_time |
> build_email | build_time |
> +--+---+-++++
> | 1.4.0| 32b871b24c7b69f59a1d2e70f444eed6e599e825  | 
> [maven-release-plugin] prepare release drill-1.4.0  | 08.12.2015 @ 00:24:59 
> PST  | venki.koruka...@gmail.com  | 08.12.2015 @ 01:14:39 PST  |
> +--+---+-++++
> 0: jdbc:drill:zk=local> select * from sys.options where status <> 'DEFAULT';
> +-+---+-+--+--+-+---++
> |name | kind  |  type   |  status  | num_val  | 
> string_val  | bool_val  | float_val  |
> +-+---+-+--+--+-+---++
> | planner.slice_target| LONG  | SYSTEM  | CHANGED  | 10   | null  
>   | null  | null   |
> | planner.width.max_per_node  | LONG  | SYSTEM  | CHANGED  | 5| null  
>   | null  | null   |
> +-+---+-+--+--+-+---++
> 2 rows selected (0.16 seconds)
>Reporter: jean-claude
>Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>
> copy the parquet files in the samples directory so that you have a 12 or so
> $ ls -lha /apache-drill-1.4.0/sample-data/nationsMF/
> nationsMF1.parquet
> nationsMF2.parquet
> nationsMF3.parquet
> create a file with a few thousand lines like these
> select * from dfs.`/Users/jccote/apache-drill-1.4.0/sample-data/nationsMF` 
> limit 500;
> start drill
> $ /apache-drill-1.4.0/bin/drill-embeded
> reduce the slice target size to force drill to use multiple fragment/threads
> jdbc:drill:zk=local> system set planner.slice_target=10;
> now run the list of queries from the file your created above
> jdbc:drill:zk=local> !run /Users/jccote/test-memory-leak-using-limit.sql
> the java heap space keeps going up until the old space is at 100% and 
> eventually you get an OutOfMemoryException in drill
> $ jstat -gccause 86850 5s
>   S0 S1 E  O  M CCSYGC YGCTFGCFGCT 
> GCTLGCC GCC 
>   0.00   0.00 100.00 100.00  98.56  96.71   2279   26.682   240  458.139  
> 484.821 GCLocker Initiated GC Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   242  461.347  
> 488.028 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   245  466.630  
> 493.311 Allocation Failure   Ergonomics  
>   0.00   0.00 100.00  99.99  98.56  96.71   2279   26.682   247  470.020  
> 496.702 Allocation Failure   Ergonomics  
> If you do the same test but do not use the LIMIT then the memory usage does 
> not go up.
> If you add a where clause so that no results are returned, then the memory 
> 

[jira] [Commented] (DRILL-4255) SELECT DISTINCT query over JSON data returns UNSUPPORTED OPERATION

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126582#comment-15126582
 ] 

Khurram Faraaz commented on DRILL-4255:
---

I have not tried that. I will test with that option set to true and share 
results here.

> SELECT DISTINCT query over JSON data returns UNSUPPORTED OPERATION
> --
>
> Key: DRILL-4255
> URL: https://issues.apache.org/jira/browse/DRILL-4255
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: CentOS
>Reporter: Khurram Faraaz
>
> SELECT DISTINCT over mapr fs generated audit logs (JSON files) results in 
> unsupported operation. An exact query over another set of JSON data returns 
> correct results.
> MapR Drill 1.4.0, commit ID : 9627a80f
> MapRBuildVersion : 5.1.0.36488.GA
> OS : CentOS x86_64 GNU/Linux
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select distinct t.operation from `auditlogs` t;
> Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema 
> changes
> Fragment 3:3
> [Error Id: 1233bf68-13da-4043-a162-cf6d98c07ec9 on example.com:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-01-08 11:35:35,093 [297060f9-1c7a-b32c-09e8-24b5ad863e73:frag:3:3] INFO  
> o.a.d.e.p.i.aggregate.HashAggBatch - User Error Occurred
> org.apache.drill.common.exceptions.UserException: UNSUPPORTED_OPERATION 
> ERROR: Hash aggregate does not support schema changes
> [Error Id: 1233bf68-13da-4043-a162-cf6d98c07ec9 ]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
>  ~[drill-common-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:144)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) 
> [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) 
> [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at java.security.AccessController.doPrivileged(Native Method) 
> [na:1.7.0_65]
> at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_65]
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
>  [hadoop-common-2.7.0-mapr-1506.jar:na]
>  at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
>  [drill-java-exec-1.4.0.jar:1.4.0]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.4.0.jar:1.4.0]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> {noformat}
> Query plan for above query.
> {noformat}
> 00-00Screen : rowType = RecordType(ANY operation): rowcount = 141437.16, 
> cumulative cost = {3.4100499276E7 rows, 1.69455861396E8 cpu, 0.0 io, 
> 1.2165858754560001E10 network, 2.738223417605E8 memory}, id = 7572
> 00-01  UnionExchange : rowType = RecordType(ANY operation): rowcount = 
> 141437.16, cumulative cost = {3.408635556E7 rows, 1.6944171768E8 cpu, 0.0 io, 
> 1.2165858754560001E10 network, 2.738223417605E8 memory}, id = 7571
> 01-01Project(operation=[$0]) : 

[jira] [Commented] (DRILL-4249) DROP TABLE HANGS indefinitely

[Sudheesh Katkam (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126593#comment-15126593
 ] 

Sudheesh Katkam commented on DRILL-4249:


The output does not state there is a deadlock. It seems there is an error while 
trying to _detect deadlocks_.

> DROP TABLE HANGS indefinitely
> -
>
> Key: DRILL-4249
> URL: https://issues.apache.org/jira/browse/DRILL-4249
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>
> DROP TABLE hangs. Table was created using CTAS and the input to CTAS was a 
> directory that had 2000 JSON files.
> There is no Exception or error message on sqlline prompt nor in drillbit.log
> Drill 1.4.0, git commitID: 32b85160
> hadoop fs -ls /tmp shows that the directory exists (at the time when DROP 
> TABLE was hung)
> drwxr-xr-x   - mapr mapr   10550677 2016-01-05 12:14 /tmp/tblMD332_wp
> {noformat}
> Snippet from drillbit.log, there's no further information after the below 
> text in drillbit.log
> 2016-01-06 10:03:09,817 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 297319a1-fb9d-8247-7d12-c8d551d56d2d: drop table tblMD332_wp
> 2016-01-06 10:03:11,516 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 6 ms
> 2016-01-06 10:03:11,517 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> {noformat}
> {noformat}
> Note that DROP TABLE commands hangs indefinitely.
> [root@centos-01 bin]# ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> apache drill 1.4.0
> "got drill?"
> 0: jdbc:drill:schema=dfs.tmp> drop table tblMD332_wp;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4329) 50 Unit tests are failing with JDK 8

[Deneche A. Hakim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deneche A. Hakim updated DRILL-4329:

Issue Type: Bug  (was: Sub-task)
Parent: (was: DRILL-1491)

> 50 Unit tests are failing with JDK 8
> 
>
> Key: DRILL-4329
> URL: https://issues.apache.org/jira/browse/DRILL-4329
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
> Environment: Mac OS
> JDK 1.8.0_65
>Reporter: Deneche A. Hakim
> Fix For: Future
>
>
> The following unit tests are failing when building Drill with JDK 1.8.0_65:
> {noformat}
>   TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject
>   TestFrameworkTest {
>   testRepeatedColumnMatching
>   testCSVVerificationOfOrder_checkFailure
>   }
>   Drill2489CallsAfterCloseThrowExceptionsTest {
> testClosedDatabaseMetaDataMethodsThrowRight
> testClosedPlainStatementMethodsThrowRight
> testclosedPreparedStmtOfOpenConnMethodsThrowRight
> testClosedResultSetMethodsThrowRight1
> testClosedResultSetMethodsThrowRight2
>   }
>   Drill2769UnsupportedReportsUseSqlExceptionTest {
> testPreparedStatementMethodsThrowRight
> testPlainStatementMethodsThrowRight
>   }
>   TestMongoFilterPushDown {
> testFilterPushDownIsEqual
> testFilterPushDownGreaterThanWithSingleField
> testFilterPushDownLessThanWithSingleField
>   }
> TestHiveStorage {
> orderByOnHiveTable
> queryingTableWithSerDeInHiveContribJar
> queryingTablesInNonDefaultFS
> readFromAlteredPartitionedTable
> readAllSupportedHiveDataTypesNativeParquet
> queryingHiveAvroTable
> readAllSupportedHiveDataTypes
> convertFromOnHiveBinaryType
>   }
>   TestInfoSchemaOnHiveStorage {
> showDatabases
> showTablesFromDb
> defaultSchemaHive
> defaultTwoLevelSchemaHive
> describeTable1
> describeTable2
> describeTable3
> describeTable4
> describeTable5
> describeTable6
> describeTable7
> describeTable8
> describeTable9
> varCharMaxLengthAndDecimalPrecisionInInfoSchema
>   }   
>   TestSqlStdBasedAuthorization {
> showSchemas
> showTables_user0
> showTables_user1
>   }
>   TestStorageBasedHiveAuthorization {
> showSchemas
> showTablesUser0
> showTablesUser1
> showTablesUser2
>   }
>   TestViewSupportOnHiveTables {
> viewWithSelectFieldsInDef_StarInQuery
> testInfoSchemaWithHiveView
> viewWithStarInDef_SelectFieldsInQuery1
> viewWithStarInDef_SelectFieldsInQuery2
> viewWithSelectFieldsInDef_SelectFieldsInQuery
> viewWithStarInDef_StarInQuery
>   }
>   TestGeometryFunctions {
> testGeometryPointCreation
> testGeometryFromTextCreation
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4331) TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject fail in Java 8

[Deneche A. Hakim (JIRA)]

Deneche A. Hakim created DRILL-4331:
---

 Summary: 
TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject fail in Java 8
 Key: DRILL-4331
 URL: https://issues.apache.org/jira/browse/DRILL-4331
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Tools, Build & Test
Affects Versions: 1.5.0
Reporter: Deneche A. Hakim


This test expects the following Project in the query plan:
{noformat}
Project(EXPR$0=[$1], rownum=[$0])
{noformat}

In Java 8, for some reason the scan operator exposes the columns in reverse 
order which causes the project to be different than the one expected:
{noformat}
Project(EXPR$0=[$0], rownum=[$1])
{noformat}

The plan is still correct, so the test must be fixed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4333) tests in Drill2489CallsAfterCloseThrowExceptionsTest fail in Java 8

[Deneche A. Hakim (JIRA)]

Deneche A. Hakim created DRILL-4333:
---

 Summary: tests in Drill2489CallsAfterCloseThrowExceptionsTest fail 
in Java 8
 Key: DRILL-4333
 URL: https://issues.apache.org/jira/browse/DRILL-4333
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Tools, Build & Test
Affects Versions: 1.5.0
Reporter: Deneche A. Hakim


The following tests fail in Java 8:
{noformat}
Drill2489CallsAfterCloseThrowExceptionsTest.testClosedPlainStatementMethodsThrowRight
Drill2489CallsAfterCloseThrowExceptionsTest.testclosedPreparedStmtOfOpenConnMethodsThrowRight
Drill2489CallsAfterCloseThrowExceptionsTest.testClosedResultSetMethodsThrowRight1
Drill2489CallsAfterCloseThrowExceptionsTest.testClosedResultSetMethodsThrowRight2
Drill2489CallsAfterCloseThrowExceptionsTest.testClosedDatabaseMetaDataMethodsThrowRight
Drill2769UnsupportedReportsUseSqlExceptionTest.testPreparedStatementMethodsThrowRight
Drill2769UnsupportedReportsUseSqlExceptionTest.testPlainStatementMethodsThrowRight
{noformat}

Drill has special implementations of Statement, PreparedStatement, ResultSet 
and DatabaseMetadata that overrides all parent methods to make sure they throw 
a proper exception if the statement has already been closed. 
These tests use reflection to make sure all methods behave correctly, but Java 
8 added more methods that need to be properly overridden.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4249) DROP TABLE HANGS indefinitely

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126579#comment-15126579
 ] 

Khurram Faraaz commented on DRILL-4249:
---

Can some one please take a look at the stacktrace and confirm if this is a 
Drill issue or it is a filesystem issue ?

> DROP TABLE HANGS indefinitely
> -
>
> Key: DRILL-4249
> URL: https://issues.apache.org/jira/browse/DRILL-4249
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>
> DROP TABLE hangs. Table was created using CTAS and the input to CTAS was a 
> directory that had 2000 JSON files.
> There is no Exception or error message on sqlline prompt nor in drillbit.log
> Drill 1.4.0, git commitID: 32b85160
> hadoop fs -ls /tmp shows that the directory exists (at the time when DROP 
> TABLE was hung)
> drwxr-xr-x   - mapr mapr   10550677 2016-01-05 12:14 /tmp/tblMD332_wp
> {noformat}
> Snippet from drillbit.log, there's no further information after the below 
> text in drillbit.log
> 2016-01-06 10:03:09,817 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 297319a1-fb9d-8247-7d12-c8d551d56d2d: drop table tblMD332_wp
> 2016-01-06 10:03:11,516 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 6 ms
> 2016-01-06 10:03:11,517 [297319a1-fb9d-8247-7d12-c8d551d56d2d:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> {noformat}
> {noformat}
> Note that DROP TABLE commands hangs indefinitely.
> [root@centos-01 bin]# ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> apache drill 1.4.0
> "got drill?"
> 0: jdbc:drill:schema=dfs.tmp> drop table tblMD332_wp;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4332) tests in TestFrameworkTest fail in Java 8

[Deneche A. Hakim (JIRA)]

Deneche A. Hakim created DRILL-4332:
---

 Summary: tests in TestFrameworkTest fail in Java 8
 Key: DRILL-4332
 URL: https://issues.apache.org/jira/browse/DRILL-4332
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Tools, Build & Test
Affects Versions: 1.5.0
Reporter: Deneche A. Hakim


the following unit tests fail in Java 8:
{noformat}
TestFrameworkTest.testRepeatedColumnMatching
TestFrameworkTest.testCSVVerificationOfOrder_checkFailure
{noformat}

The tests expect the query to fail with a specific error message. The message 
generated by DrillTestWrapper.compareMergedVectors assumes a specific order in 
a map keySet (which we shouldn't). In Java 8 it seems the order changed which 
causes a slightly different error message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4330) Long running SQL query hangs once Foreman node is killed

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4330:
-

 Summary: Long running SQL query hangs once Foreman node is killed
 Key: DRILL-4330
 URL: https://issues.apache.org/jira/browse/DRILL-4330
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.4.0
 Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz


Summary : Once Foreman node Drillbit is killed, long running query just hangs 
and no profile information is written to Web UI. That long running query was 
issued from the Foreman node.

MapR Drill 1.4.0 GA
MapR FS 5.0.0 GA
JDK8
4 node CentOS cluster

./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
Issue a long running select query over JSON data
Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill 
-9 PID

The long running query hangs on sqlline prompt, there are no 
messages/errors/Exceptions reported on sqlline prompt.

On the Web UI there is no profile reported for the long running query that was 
running on the Drillbit that was killed.

Question (1) : Why was there no profile reported/written on the Web UI for that 
long running query ? In a real production scenario user will not know what 
query was under execution at the point when Foreman went down. 

Question (2) : Why does the long running query not terminate, once the foreman 
was killed ? from the drillbit.log snippet we do not see any 
CANCELED/TERMINATED message for that query, why ?

Snippet from drillbit.log on the foreman node. 
{noformat}
2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 18.28 μs, Latest start: 18.28 μs, Average 
start: 18.28 μs .
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: 
State to report: RUNNING
{noformat}

Doing kill -3 PID on the non foreman node for the Drillbit process gives us 
stack trace in drillbit.out
{noformat}
2016-02-01 11:03:31
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):

"qtp801808302-129" #129 prio=5 os_prio=0 tid=0x7f7ad8127000 nid=0xaad 
waiting on condition [0x7f7ab62fb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007b69100a8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 

[jira] [Updated] (DRILL-4330) Long running SQL query hangs once Foreman node is killed

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-4330:
--
Attachment: drillbit.out

Added drillbit.out file , it has stack trace

> Long running SQL query hangs once Foreman node is killed
> 
>
> Key: DRILL-4330
> URL: https://issues.apache.org/jira/browse/DRILL-4330
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
> Attachments: drillbit.out
>
>
> Summary : Once Foreman node Drillbit is killed, long running query just hangs 
> and no profile information is written to Web UI. That long running query was 
> issued from the Foreman node.
> MapR Drill 1.4.0 GA
> MapR FS 5.0.0 GA
> JDK8
> 4 node CentOS cluster
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> Issue a long running select query over JSON data
> Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill 
> -9 PID
> The long running query hangs on sqlline prompt, there are no 
> messages/errors/Exceptions reported on sqlline prompt.
> On the Web UI there is no profile reported for the long running query that 
> was running on the Drillbit that was killed.
> Question (1) : Why was there no profile reported/written on the Web UI for 
> that long running query ? In a real production scenario user will not know 
> what query was under execution at the point when Foreman went down. 
> Question (2) : Why does the long running query not terminate, once the 
> foreman was killed ? from the drillbit.log snippet we do not see any 
> CANCELED/TERMINATED message for that query, why ?
> Snippet from drillbit.log on the foreman node. 
> {noformat}
> 2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
> 2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Earliest start: 18.28 μs, Latest start: 18.28 μs, 
> Average start: 18.28 μs .
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
> {noformat}
> Doing kill -3 PID on the non foreman node for the Drillbit process gives us 
> stack trace in drillbit.out
> {noformat}
> 2016-02-01 11:03:31
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-129" #129 prio=5 os_prio=0 tid=0x7f7ad8127000 nid=0xaad 
> waiting on condition [0x7f7ab62fb000]
>java.lang.Thread.State: TIMED_WAITING 

[jira] [Commented] (DRILL-4329) 50 Unit tests are failing with JDK 8

[Jacques Nadeau (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126565#comment-15126565
 ] 

Jacques Nadeau commented on DRILL-4329:
---

Can you provide more details of the error types/failures?

> 50 Unit tests are failing with JDK 8
> 
>
> Key: DRILL-4329
> URL: https://issues.apache.org/jira/browse/DRILL-4329
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
> Environment: Mac OS
> JDK 1.8.0_65
>Reporter: Deneche A. Hakim
> Fix For: Future
>
>
> The following unit tests are failing when building Drill with JDK 1.8.0_65:
> {noformat}
>   TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject
>   TestFrameworkTest {
>   testRepeatedColumnMatching
>   testCSVVerificationOfOrder_checkFailure
>   }
>   Drill2489CallsAfterCloseThrowExceptionsTest {
> testClosedDatabaseMetaDataMethodsThrowRight
> testClosedPlainStatementMethodsThrowRight
> testclosedPreparedStmtOfOpenConnMethodsThrowRight
> testClosedResultSetMethodsThrowRight1
> testClosedResultSetMethodsThrowRight2
>   }
>   Drill2769UnsupportedReportsUseSqlExceptionTest {
> testPreparedStatementMethodsThrowRight
> testPlainStatementMethodsThrowRight
>   }
>   TestMongoFilterPushDown {
> testFilterPushDownIsEqual
> testFilterPushDownGreaterThanWithSingleField
> testFilterPushDownLessThanWithSingleField
>   }
> TestHiveStorage {
> orderByOnHiveTable
> queryingTableWithSerDeInHiveContribJar
> queryingTablesInNonDefaultFS
> readFromAlteredPartitionedTable
> readAllSupportedHiveDataTypesNativeParquet
> queryingHiveAvroTable
> readAllSupportedHiveDataTypes
> convertFromOnHiveBinaryType
>   }
>   TestInfoSchemaOnHiveStorage {
> showDatabases
> showTablesFromDb
> defaultSchemaHive
> defaultTwoLevelSchemaHive
> describeTable1
> describeTable2
> describeTable3
> describeTable4
> describeTable5
> describeTable6
> describeTable7
> describeTable8
> describeTable9
> varCharMaxLengthAndDecimalPrecisionInInfoSchema
>   }   
>   TestSqlStdBasedAuthorization {
> showSchemas
> showTables_user0
> showTables_user1
>   }
>   TestStorageBasedHiveAuthorization {
> showSchemas
> showTablesUser0
> showTablesUser1
> showTablesUser2
>   }
>   TestViewSupportOnHiveTables {
> viewWithSelectFieldsInDef_StarInQuery
> testInfoSchemaWithHiveView
> viewWithStarInDef_SelectFieldsInQuery1
> viewWithStarInDef_SelectFieldsInQuery2
> viewWithSelectFieldsInDef_SelectFieldsInQuery
> viewWithStarInDef_StarInQuery
>   }
>   TestGeometryFunctions {
> testGeometryPointCreation
> testGeometryFromTextCreation
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4330) Long running SQL query hangs once Foreman node is killed

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126577#comment-15126577
 ] 

Khurram Faraaz commented on DRILL-4330:
---

Adding Hakim's comment from email thread on this JIRA.


I don't see any fragment running which means the query did cancel itself 
properly, yet some threads are blocked. Please report this in the JIRA too.


> Long running SQL query hangs once Foreman node is killed
> 
>
> Key: DRILL-4330
> URL: https://issues.apache.org/jira/browse/DRILL-4330
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
> Attachments: drillbit.out
>
>
> Summary : Once Foreman node Drillbit is killed, long running query just hangs 
> and no profile information is written to Web UI. That long running query was 
> issued from the Foreman node.
> MapR Drill 1.4.0 GA
> MapR FS 5.0.0 GA
> JDK8
> 4 node CentOS cluster
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> Issue a long running select query over JSON data
> Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill 
> -9 PID
> The long running query hangs on sqlline prompt, there are no 
> messages/errors/Exceptions reported on sqlline prompt.
> On the Web UI there is no profile reported for the long running query that 
> was running on the Drillbit that was killed.
> Question (1) : Why was there no profile reported/written on the Web UI for 
> that long running query ? In a real production scenario user will not know 
> what query was under execution at the point when Foreman went down. 
> Question (2) : Why does the long running query not terminate, once the 
> foreman was killed ? from the drillbit.log snippet we do not see any 
> CANCELED/TERMINATED message for that query, why ?
> Snippet from drillbit.log on the foreman node. 
> {noformat}
> 2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
> 2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Earliest start: 18.28 μs, Latest start: 18.28 μs, 
> Average start: 18.28 μs .
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
> {noformat}
> Doing kill -3 PID on the non foreman node for the Drillbit process gives us 
> stack trace in drillbit.out
> {noformat}
> 2016-02-01 11:03:31
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed 

[jira] [Commented] (DRILL-4255) SELECT DISTINCT query over JSON data returns UNSUPPORTED OPERATION

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126597#comment-15126597
 ] 

Khurram Faraaz commented on DRILL-4255:
---

Here are the test results when we set `store.json.all_text_mode` is set to 
true, we see an IllegalStateException being returned

{noformat}
0: jdbc:drill:schema=dfs.tmp> select distinct t.operation from `auditlogs` t;
Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema 
changes

Fragment 1:0

[Error Id: ca0ffebd-4cee-41ba-9b56-c637128516a4 on centos-02.qa.lab:31010] 
(state=,code=0)

# Set option to true

0: jdbc:drill:schema=dfs.tmp> alter session set  
`store.json.all_text_mode`=true;
+---++
|  ok   |  summary   |
+---++
| true  | store.json.all_text_mode updated.  |
+---++
1 row selected (0.213 seconds)

0: jdbc:drill:schema=dfs.tmp> select distinct t.operation from `auditlogs` t;
Error: SYSTEM ERROR: IllegalStateException: Failure while reading vector.  
Expected vector class of org.apache.drill.exec.vector.NullableIntVector but was 
holding vector class org.apache.drill.exec.vector.NullableVarCharVector.

Fragment 2:0

[Error Id: eb354a62-235f-47a4-a112-0e8f45758d9e on centos-02.qa.lab:31010] 
(state=,code=0)
{noformat}

{noformat}
2016-02-01 17:16:04,423 [29506d2e-577e-8bbd-3b8d-fd2d97fabb70:frag:2:0] ERROR 
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
Failure while reading vector.  Expected vector class of 
org.apache.drill.exec.vector.NullableIntVector but was holding vector class 
org.apache.drill.exec.vector.NullableVarCharVector.

Fragment 2:0

[Error Id: eb354a62-235f-47a4-a112-0e8f45758d9e on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
IllegalStateException: Failure while reading vector.  Expected vector class of 
org.apache.drill.exec.vector.NullableIntVector but was holding vector class 
org.apache.drill.exec.vector.NullableVarCharVector.

Fragment 2:0

[Error Id: eb354a62-235f-47a4-a112-0e8f45758d9e on centos-02.qa.lab:31010]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
 ~[drill-common-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
 [drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
 [drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
 [drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) 
[drill-common-1.4.0.jar:1.4.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_65]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.lang.IllegalStateException: Failure while reading vector.  
Expected vector class of org.apache.drill.exec.vector.NullableIntVector but was 
holding vector class org.apache.drill.exec.vector.NullableVarCharVector.
at 
org.apache.drill.exec.record.VectorContainer.getValueAccessorById(VectorContainer.java:284)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.record.RecordBatchLoader.getValueAccessorById(RecordBatchLoader.java:179)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.physical.impl.unorderedreceiver.UnorderedReceiverBatch.getValueAccessorById(UnorderedReceiverBatch.java:135)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.getValueAccessorById(IteratorValidatorBatchIterator.java:188)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.test.generated.PartitionerGen11$OutgoingRecordBatch.doSetup(PartitionerTemplate.java:61)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.PartitionerGen11$OutgoingRecordBatch.initializeBatch(PartitionerTemplate.java:358)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.PartitionerGen11.flushOutgoingBatches(PartitionerTemplate.java:163)
 ~[na:na]
at 
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator$FlushBatchesHandlingClass.execute(PartitionerDecorator.java:266)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.executeMethodLogic(PartitionerDecorator.java:138)
 ~[drill-java-exec-1.4.0.jar:1.4.0]
at 
org.apache.drill.exec.physical.impl.partitionsender.PartitionerDecorator.flushOutgoingBatches(PartitionerDecorator.java:82)
 

[jira] [Created] (DRILL-4334) tests in TestMongoFilterPushDown fail in Java 8

[Deneche A. Hakim (JIRA)]

Deneche A. Hakim created DRILL-4334:
---

 Summary: tests in TestMongoFilterPushDown fail in Java 8
 Key: DRILL-4334
 URL: https://issues.apache.org/jira/browse/DRILL-4334
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Tools, Build & Test
Affects Versions: 1.5.0
Reporter: Deneche A. Hakim


All tests in TestMongoFilterPushDown fail in Java8. It looks like the filter is 
not pushed down to the Mongo storage plugin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4338) Concurrent query remains in CANCELLATION_REQUESTED state

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4338:
-

 Summary: Concurrent query remains in CANCELLATION_REQUESTED state 
 Key: DRILL-4338
 URL: https://issues.apache.org/jira/browse/DRILL-4338
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.4.0
 Environment: 4 node cluster CentOS
Reporter: Khurram Faraaz


Execute a query concurrently through a Java program and while the java program 
is under execution (executing SQL queries concurrently) issue Ctrl-C on the 
prompt where the java program was being executed.

Here are two observations, 
(1) There is an Exception in drillbit.log.
(2) Once Ctrl-C was issued to the java program, queries that were under 
execution at that point of time, move from FAILED state to 
CANCELLATION_REQUESTED state, they do not end up in CANCELED state. Ideally 
that last state of these queries should be CANCELED state and not 
CANCELLATION_REQUESTED. 

Snippet from drillbit.log
{noformat}
2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: 
State change requested AWAITING_ALLOCATION --> RUNNING
2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  
o.a.d.e.w.f.FragmentStatusReporter - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: 
State to report: RUNNING
2016-02-02 06:21:48,560 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler 
- Exception in RPC communication.  Connection: /10.10.100.201:31010 <--> 
/10.10.100.201:45087 (user client).  Closing connection.
java.io.IOException: syscall:read(...)() failed: Connection reset by peer
2016-02-02 06:21:48,562 [UserServer-1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: 
State change requested RUNNING --> FAILED
2016-02-02 06:21:48,562 [UserServer-1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: 
State change requested RUNNING --> FAILED
2016-02-02 06:21:48,562 [UserServer-1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: 
State change requested RUNNING --> FAILED
2016-02-02 06:21:48,563 [UserServer-1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: 
State change requested RUNNING --> FAILED
2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: 
State change requested FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] WARN  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: 
Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: 
State change requested FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] WARN  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: 
Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: 
State change requested FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] WARN  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: 
Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: 
State change requested FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] WARN  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: 
Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED
2016-02-02 06:21:48,597 [UserServer-1] INFO  
o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: 
State change requested FAILED --> FAILED
2016-02-02 06:21:48,599 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler 
- Exception occurred with closed channel.  Connection: /10.10.100.201:31010 
<--> /10.10.100.201:45087 (user client)
io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one 
message.
at 
io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
 ~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
 [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
 [netty-transport-4.0.27.Final.jar:4.0.27.Final]
at 

[jira] [Resolved] (DRILL-4150) AssertionError: no provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop provider

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu resolved DRILL-4150.
--
Resolution: Cannot Reproduce

> AssertionError: no provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], 
> m=interface org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a 
> backstop provider is recommended
> 
>
> Key: DRILL-4150
> URL: https://issues.apache.org/jira/browse/DRILL-4150
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>
> Test fails with AssertionError on Drill 1.4 with assertions enabled and JDK8, 
> on 4 node cluster. Test was executed as part of functional test run on 4 node 
> cluster.
> Assertion seems to be coming from calcite.
> Failing test case : Functional/decimal/decimal101.q
> {code}
> query => select cast('0.0001' as decimal(9,9)) / 
> cast('-' as decimal(28,0)) from data limit 1;
> {code}
> {noformat}
> git.commit.id=ff76078b61a54001da1de3dbd95d01004da4b791
> [root@centos-01 drill-output]# java -version
> openjdk version "1.8.0_65"
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2015-12-01 03:31:50,248 [29a2eb5a-785d-4e55-6148-b4fd5a5cfc9c:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: no 
> provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], m=interface 
> org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a backstop 
> provider is recommended
> [Error Id: 116e2b9c-e247-44c2-9490-ee6fd22736cc on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> AssertionError: no provider found (rel=rel#16:Subset#0.ENUMERABLE.ANY([]).[], 
> m=interface org.apache.calcite.rel.metadata.BuiltInMetadata$Collation); a 
> backstop provider is recommended
> [Error Id: 116e2b9c-e247-44c2-9490-ee6fd22736cc on centos-02.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
>  ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule ReduceExpressionsRule_Project, args 
> [rel#17:LogicalProject.NONE.ANY([]).[](input=rel#16:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=/(CAST('0.0001'):DECIMAL(9,
>  9) NOT NULL, CAST('-'):DECIMAL(28, 0) NOT NULL))]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying 
> rule ReduceExpressionsRule_Project, args 
> [rel#17:LogicalProject.NONE.ANY([]).[](input=rel#16:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=/(CAST('0.0001'):DECIMAL(9,
>  9) NOT NULL, CAST('-'):DECIMAL(28, 0) NOT NULL))]
> at org.apache.calcite.util.Util.newInternal(Util.java:792) 
> ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>  

[jira] [Resolved] (DRILL-4148) NullPointerException in planning when running window function query

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu resolved DRILL-4148.
--
Resolution: Cannot Reproduce

> NullPointerException in planning when running window function query
> ---
>
> Key: DRILL-4148
> URL: https://issues.apache.org/jira/browse/DRILL-4148
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>
> Failing test case : Functional/window_functions/lag_func/lag_Fn_4.q
> Tests were run on JDK8 with assertions enabled.
> {noformat}
> [root@centos-01 lag_func]# java -version
> openjdk version "1.8.0_65"
> OpenJDK Runtime Environment (build 1.8.0_65-b17)
> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
> [root@centos-01 lag_func]# uname -a
> Linux centos-01 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 
> x86_64 x86_64 x86_64 GNU/Linux
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2015-12-01 03:31:50,133 [29a2eb59-d5a3-f0f2-d2df-9ad4e5d17109:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: NullPointerException
> [Error Id: ab046c45-4a2d-428d-8c72-592a02ea53e5 on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> NullPointerException
> [Error Id: ab046c45-4a2d-428d-8c72-592a02ea53e5 on centos-02.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
>  ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
>  [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: null
> ... 4 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at 
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) 
> ~[guava-14.0.1.jar:na]
> at 
> org.apache.calcite.rel.metadata.CachingRelMetadataProvider$CachingInvocationHandler.(CachingRelMetadataProvider.java:104)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:78)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:75)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.plan.hep.HepRelMetadataProvider$1.apply(HepRelMetadataProvider.java:45)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.plan.hep.HepRelMetadataProvider$1.apply(HepRelMetadataProvider.java:34)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:77)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.metadata.CachingRelMetadataProvider$2.apply(CachingRelMetadataProvider.java:75)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.metadata.MetadataFactoryImpl.query(MetadataFactoryImpl.java:69)
>  ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> at 
> org.apache.calcite.rel.AbstractRelNode.metadata(AbstractRelNode.java:271) 
> ~[calcite-core-1.4.0-drill-r9.jar:1.4.0-drill-r9]
> 

[jira] [Resolved] (DRILL-4144) AssertionError : Internal error: Error while applying rule HivePushPartitionFilterIntoScan:Filter_On_Project_Hive

[Sean Hsuan-Yi Chu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Hsuan-Yi Chu resolved DRILL-4144.
--
Resolution: Cannot Reproduce

> AssertionError : Internal error: Error while applying rule 
> HivePushPartitionFilterIntoScan:Filter_On_Project_Hive
> -
>
> Key: DRILL-4144
> URL: https://issues.apache.org/jira/browse/DRILL-4144
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.3.0
> Environment: 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>
> AssertionError seen on Drill 1.3 version d61bb83a on a 4 node cluster, as 
> part of Functional test run on JDK 8. Note that assertions were enabled as 
> part of test execution.
> {code}
> [root@centos-01 bin]# java -version
> openjdk version "1.8.0_65"
> OpenJDK Runtime Environment (build 1.8.0_65-b17)
> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
> {code}
> Failing test case : 
> Functional/interpreted_partition_pruning/hive/text/hier_intint/plan/4.q
> {code}
> query => explain plan for select l_orderkey, l_partkey, l_quantity, 
> l_shipdate, l_shipinstruct from 
> hive.lineitem_text_partitioned_hive_hier_intint where case when `month` > 11 
> then 2 else null end is not null and `year` = 1991;
> {code}
> {noformat}
> [Error Id: c0e23293-2592-4421-9953-bc7d6488398f on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError
> [Error Id: c0e23293-2592-4421-9953-bc7d6488398f on centos-03.qa.lab:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
>  ~[drill-common-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
>  [drill-java-exec-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
>  [drill-java-exec-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
>  [drill-java-exec-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73) 
> [drill-common-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
>  [drill-java-exec-1.3.0.jar:1.3.0]
>   at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894) 
> [drill-java-exec-1.3.0.jar:1.3.0]
>   at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255) 
> [drill-java-exec-1.3.0.jar:1.3.0]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: Error while 
> applying rule HivePushPartitionFilterIntoScan:Filter_On_Project_Hive, args 
> [rel#879659:DrillFilterRel.LOGICAL.ANY([]).[](input=rel#879658:Subset#8.LOGICAL.ANY([]).[],condition=AND(IS
>  NOT NULL(CASE(>($6, 11), 2, null)), =($5, 1991))), 
> rel#879657:DrillProjectRel.LOGICAL.ANY([]).[](input=rel#879656:Subset#7.LOGICAL.ANY([]).[],l_orderkey=$6,l_partkey=$3,l_quantity=$0,l_shipdate=$5,l_shipinstruct=$1,year=$2,month=$4),
>  rel#879639:DrillScanRel.LOGICAL.ANY([]).[](table=[hive, 
> lineitem_text_partitioned_hive_hier_intint],groupscan=HiveScan 
> [table=Table(dbName:default, 
> tableName:lineitem_text_partitioned_hive_hier_intint), 
> inputSplits=[maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/1/lineitemaa.tbl:0+106992,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/10/lineitemaj.tbl:0+106646,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/11/lineitemak.tbl:0+106900,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/12/lineitemal.tbl:0+11926,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/2/lineitemab.tbl:0+106663,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/3/lineitemac.tbl:0+106980,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/4/lineitemad.tbl:0+106276,
>  
> maprfs:///drill/testdata/partition_pruning/hive/text/lineitem_hierarchical_intint/1991/5/lineitemae.tbl:0+107315,
>  
> 

[jira] [Updated] (DRILL-4338) Concurrent query remains in CANCELLATION_REQUESTED state

[Khurram Faraaz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-4338:
--
Attachment: query_In_cancellation_requested_state.png
ConcurrencyTest.java

Added concurrency test source here.
Also added screen shot of queries from Drill UI that shows that queries are in 
CANCELLATION_REQUESTED state.

> Concurrent query remains in CANCELLATION_REQUESTED state 
> -
>
> Key: DRILL-4338
> URL: https://issues.apache.org/jira/browse/DRILL-4338
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
> Attachments: ConcurrencyTest.java, 
> query_In_cancellation_requested_state.png
>
>
> Execute a query concurrently through a Java program and while the java 
> program is under execution (executing SQL queries concurrently) issue Ctrl-C 
> on the prompt where the java program was being executed.
> Here are two observations, 
> (1) There is an Exception in drillbit.log.
> (2) Once Ctrl-C was issued to the java program, queries that were under 
> execution at that point of time, move from FAILED state to 
> CANCELLATION_REQUESTED state, they do not end up in CANCELED state. Ideally 
> that last state of these queries should be CANCELED state and not 
> CANCELLATION_REQUESTED. 
> Snippet from drillbit.log
> {noformat}
> 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State to report: RUNNING
> 2016-02-02 06:21:48,560 [UserServer-1] ERROR 
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
> Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client).  
> Closing connection.
> java.io.IOException: syscall:read(...)() failed: Connection reset by peer
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested RUNNING --> 
> FAILED
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested RUNNING --> 
> FAILED
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested RUNNING --> 
> FAILED
> 2016-02-02 06:21:48,563 [UserServer-1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested RUNNING --> 
> FAILED
> 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> 
> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-f424-6adc-d668-1659e4353698:0:0: Ignoring unexpected state 
> transition FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested FAILED --> 
> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: Ignoring unexpected state 
> transition FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested FAILED --> 
> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: Ignoring unexpected state 
> transition FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested FAILED --> 
> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] WARN  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: Ignoring unexpected state 
> transition FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,597 [UserServer-1] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> 
> FAILED
> 2016-02-02 06:21:48,599 [UserServer-1] WARN  
> o.a.d.exec.rpc.RpcExceptionHandler - 

[jira] [Commented] (DRILL-4287) Do lazy reading of parquet metadata cache file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127594#comment-15127594
 ] 

ASF GitHub Bot commented on DRILL-4287:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/345#discussion_r51518520
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -118,13 +133,34 @@ public boolean apply(@Nullable FileStatus status) {
   }
 }));
 
-return create(nonDirectories, null, selectionRoot);
+final FileSelection fileSel = create(nonDirectories, null, 
selectionRoot);
--- End diff --

Does it make sense to pass in the list of Files in addition to the list of 
FileStatus? I thought the implicit assumption for FileSelection is list of 
files goes first before FileStatus. 


> Do lazy reading of parquet metadata cache file
> --
>
> Key: DRILL-4287
> URL: https://issues.apache.org/jira/browse/DRILL-4287
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4128) null pointer at org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127611#comment-15127611
 ] 

ASF GitHub Bot commented on DRILL-4128:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/353


> null pointer at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> -
>
> Key: DRILL-4128
> URL: https://issues.apache.org/jira/browse/DRILL-4128
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
>Reporter: Devender Yadav 
>Priority: Blocker
> Fix For: Future
>
>
> While fetching data from ResultSet in JDBC. I got the null pointer. Details - 
> java.lang.NullPointerException
> at 
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> at 
> org.apache.drill.exec.vector.accessor.BoundCheckingAccessor.getString(BoundCheckingAccessor.java:124)
> at 
> org.apache.drill.jdbc.impl.TypeConvertingSqlAccessor.getString(TypeConvertingSqlAccessor.java:649)
> at 
> org.apache.drill.jdbc.impl.AvaticaDrillSqlAccessor.getString(AvaticaDrillSqlAccessor.java:95)
> at 
> net.hydromatic.avatica.AvaticaResultSet.getString(AvaticaResultSet.java:205)
> at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.getString(DrillResultSetImpl.java:182)
> Below mentioned method is throwing null pointer becaue getObject(rowOffset) 
> returns null for null values & null.toString() is throwing null pointer.
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset).toString();
>   }
> It should be like:
>  @Override
>   public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset)==null? null:getObject(rowOffset).toString();
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4287) Do lazy reading of parquet metadata cache file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127739#comment-15127739
 ] 

ASF GitHub Bot commented on DRILL-4287:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/345#discussion_r51528367
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -118,13 +133,34 @@ public boolean apply(@Nullable FileStatus status) {
   }
 }));
 
-return create(nonDirectories, null, selectionRoot);
+final FileSelection fileSel = create(nonDirectories, null, 
selectionRoot);
--- End diff --

I did not change the existing create() call .. it was passing null for the 
list of files.  I think it is ok because FileStatus contains the Path which 
reflects the file name.  It also seems that if either the list of files is null 
or the list of statuses is null, the functions getFiles() or getStatuses() can 
compute them on the first invocation and save for future use.  


> Do lazy reading of parquet metadata cache file
> --
>
> Key: DRILL-4287
> URL: https://issues.apache.org/jira/browse/DRILL-4287
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4313) C++ client - Improve method of drillbit selection from cluster

[Jason Altekruse (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Altekruse updated DRILL-4313:
---
Fix Version/s: 1.5.0

> C++ client - Improve method of drillbit selection from cluster
> --
>
> Key: DRILL-4313
> URL: https://issues.apache.org/jira/browse/DRILL-4313
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Parth Chandra
> Fix For: 1.5.0
>
>
> The current C++ client handles multiple parallel queries over the same 
> connection, but that creates a bottleneck as the queries get sent to the same 
> drillbit.
> The client can manage this more effectively by choosing from a configurable 
> pool of connections and round robin queries to them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4287) Do lazy reading of parquet metadata cache file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127743#comment-15127743
 ] 

ASF GitHub Bot commented on DRILL-4287:
---

Github user amansinha100 commented on a diff in the pull request:

https://github.com/apache/drill/pull/345#discussion_r51528494
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ---
@@ -338,8 +354,14 @@ private boolean hasSingleValue(ColumnMetadata 
columnChunkMetaData) {
 return (columnChunkMetaData != null) && 
(columnChunkMetaData.hasSingleValue());
   }
 
+  @Override
--- End diff --

Yes, I suppose you encountered a failure in some tests due to this...


> Do lazy reading of parquet metadata cache file
> --
>
> Key: DRILL-4287
> URL: https://issues.apache.org/jira/browse/DRILL-4287
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4287) Do lazy reading of parquet metadata cache file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127595#comment-15127595
 ] 

ASF GitHub Bot commented on DRILL-4287:
---

Github user jinfengni commented on a diff in the pull request:

https://github.com/apache/drill/pull/345#discussion_r51518560
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ---
@@ -338,8 +354,14 @@ private boolean hasSingleValue(ColumnMetadata 
columnChunkMetaData) {
 return (columnChunkMetaData != null) && 
(columnChunkMetaData.hasSingleValue());
   }
 
+  @Override
--- End diff --

JsonIgnore is needed here? 



> Do lazy reading of parquet metadata cache file
> --
>
> Key: DRILL-4287
> URL: https://issues.apache.org/jira/browse/DRILL-4287
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: Jinfeng Ni
>
> Currently, the parquet metadata cache file is read eagerly during creation of 
> the DrillTable (as part of ParquetFormatMatcher.isReadable()).  This is not 
> desirable from performance standpoint since there are scenarios where we want 
> to do some up-front optimizations - e.g. directory-based partition pruning 
> (see DRILL-2517) or potential limit 0 optimization etc. - and in such 
> situations it is better to do lazy reading of the metadata cache file.   
> This is a placeholder to perform such delayed reading since it is needed for 
> the aforementioned optimizations.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (DRILL-4330) Long running SQL query hangs once Foreman node is killed

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong reassigned DRILL-4330:
---

Assignee: Sudheesh Katkam

[~sudheeshkatkam] - assigning to you since you had expressed interest in 
looking at this.

> Long running SQL query hangs once Foreman node is killed
> 
>
> Key: DRILL-4330
> URL: https://issues.apache.org/jira/browse/DRILL-4330
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sudheesh Katkam
> Attachments: drillbit.out
>
>
> Summary : Once Foreman node Drillbit is killed, long running query just hangs 
> and no profile information is written to Web UI. That long running query was 
> issued from the Foreman node.
> MapR Drill 1.4.0 GA
> MapR FS 5.0.0 GA
> JDK8
> 4 node CentOS cluster
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> Issue a long running select query over JSON data
> Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill 
> -9 PID
> The long running query hangs on sqlline prompt, there are no 
> messages/errors/Exceptions reported on sqlline prompt.
> On the Web UI there is no profile reported for the long running query that 
> was running on the Drillbit that was killed.
> Question (1) : Why was there no profile reported/written on the Web UI for 
> that long running query ? In a real production scenario user will not know 
> what query was under execution at the point when Foreman went down. 
> Question (2) : Why does the long running query not terminate, once the 
> foreman was killed ? from the drillbit.log snippet we do not see any 
> CANCELED/TERMINATED message for that query, why ?
> Snippet from drillbit.log on the foreman node. 
> {noformat}
> 2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
> 2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
> numFiles: 1
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  
> o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 
> using 1 threads. Earliest start: 18.28 μs, Latest start: 18.28 μs, 
> Average start: 18.28 μs .
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested 
> AWAITING_ALLOCATION --> RUNNING
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
> {noformat}
> Doing kill -3 PID on the non foreman node for the Drillbit process gives us 
> stack trace in drillbit.out
> {noformat}
> 2016-02-01 11:03:31
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-129" #129 prio=5 os_prio=0 tid=0x7f7ad8127000 nid=0xaad 
> 

[jira] [Comment Edited] (DRILL-2669) Error happening without limit clause and works with limit clause

[Oscar Morante (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121303#comment-15121303
 ] 

Oscar Morante edited comment on DRILL-2669 at 2/1/16 6:49 PM:
--

Hi, I think I'm running into the same issue.  Given some json data with this 
shape:

{code:javascript}
{ "timestamp": "2015-11-03T07:21:24.066Z"}
{code}

This query fails:
{code}
select to_timestamp(`timestamp`, '-MM-dd''T''HH:mm:ss.SSS''Z''') from 
(select * from mydatasource);

Error: SYSTEM ERROR: ExpressionParsingException: Expression has syntax error! 
line 1:38:mismatched input 'T' expecting CParen

Fragment 1:3

[Error Id: e1f0fd04-52a4-4b79-bef6-7262a88a7e28 on 10.2.26.3:31010] 
(state=,code=0)
{code}

And this one works:
{code}
select to_timestamp(`timestamp`, '-MM-dd''T''HH:mm:ss.SSS''Z''') from 
(select * from mydatasource limit 1); 
{code}


was (Author: spacepluk):
Hi, I think I'm running into the same issue.  Given some json data with this 
shape:

{code:javascript}
{ "timestamp": "2015-11-03T07:21:24.066Z"}
{code}

This query fails:
{code}
select to_timestamp(`timestamp`, '-MM-dd''T''HH:mm:ss.SSS''Z''') from 
mydatasource limit 1;

Error: SYSTEM ERROR: ExpressionParsingException: Expression has syntax error! 
line 1:38:mismatched input 'T' expecting CParen

Fragment 1:3

[Error Id: e1f0fd04-52a4-4b79-bef6-7262a88a7e28 on 10.2.26.3:31010] 
(state=,code=0)
{code}

And this one works:
{code}
select to_timestamp(`timestamp`, '-MM-dd''T''HH:mm:ss.SSS''Z''') from 
(select * from mydatasource); 
{code}

UPDATE: I copy the wrong error, it's ok now.

> Error happening without limit clause and works with limit clause
> 
>
> Key: DRILL-2669
> URL: https://issues.apache.org/jira/browse/DRILL-2669
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 0.8.0
> Environment: mapr sandbox 4.0.2
>Reporter: Sudhakar Thota
> Fix For: Future
>
>
> Perhaps this could be a bug. I get the same results.
> But the plan is very different, the UnionExchange is set up immediately after 
> the scan operation in successful case( Case 1 ), where as UnionExchange is 
> happening after scan>project (Case -2).
> Case -1.Successful case.
> {code}
> 0: jdbc:drill:> explain plan for select to_timestamp(t.t, 
> '-MM-dd''T''HH:mm:ss.SSS''Z''') FROM (select * from 
> dfs.sthota_prq.`/tstamp_test/*.parquet` limit 13015351) t;
> --+
> text  json
> --+
> 00-00 Screen
> 00-01 Project(EXPR$0=[TO_TIMESTAMP(ITEM($0, 't'), 
> '-MM-dd''T''HH:mm:ss.SSS''Z''')])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[13015351])
> 00-04 UnionExchange
> 01-01 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_2_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_1_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_0_0.parquet]],
>  selectionRoot=/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test, 
> numFiles=3, columns=[`*`]]])
> {
> "head" :
> Unknown macro: { "version" }
> ,
> {code}
> Case -2. Unsuccessful case:
> {code}
> 0: jdbc:drill:> explain plan for select to_timestamp(t.t, 
> '-MM-dd''T''HH:mm:ss.SSS''Z''') FROM (select * from 
> dfs.sthota_prq.`/tstamp_test/*.parquet` ) t;
> --+
> text  json
> --+
> 00-00 Screen
> 00-01 UnionExchange
> 01-01 Project(EXPR$0=[TO_TIMESTAMP(ITEM($0, 't'), 
> '-MM-dd''T''HH:mm:ss.SSS''Z''')])
> 01-02 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_2_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_1_0.parquet],
>  ReadEntryWithPath 
> [path=maprfs:/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test/1_0_0.parquet]],
>  selectionRoot=/mapr/demo.mapr.com/user/sthota/parquet/tstamp_test, 
> numFiles=3, columns=[`*`]]])
> {
> "head" :
> Unknown macro: { "version" }
> ,
> {code}
> {code}
> 0: jdbc:drill:> select to_timestamp(t.t, '-MM-dd''T''HH:mm:ss.SSS''Z''') 
> FROM (select * from dfs.sthota_prq.`/tstamp_test/*.parquet` limit 10) t;
> 
> EXPR$0
> 
> 2015-01-27 13:43:53.0
> 2015-01-27 13:43:49.0
> 2015-01-27 13:43:47.0
> 2015-01-27 13:43:47.0
> 2015-01-27 13:43:47.0
> 2015-01-27 13:43:45.0
> 2015-01-27 13:43:43.0
> 2015-01-27 13:43:43.0
> 2015-01-27 13:43:43.0
> 2015-01-27 13:43:39.0
> 
> 10 rows selected (1.127 seconds)
> {code}
> {code}
> 0: jdbc:drill:> select to_timestamp(t.t, '-MM-dd''T''HH:mm:ss.SSS''Z''') 
> FROM (select * from 

[jira] [Commented] (DRILL-4329) 50 Unit tests are failing with JDK 8

[Deneche A. Hakim (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126587#comment-15126587
 ] 

Deneche A. Hakim commented on DRILL-4329:
-

Yes, I'm investigating the failures and creating sub tasks with a general 
description of each failure

> 50 Unit tests are failing with JDK 8
> 
>
> Key: DRILL-4329
> URL: https://issues.apache.org/jira/browse/DRILL-4329
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
> Environment: Mac OS
> JDK 1.8.0_65
>Reporter: Deneche A. Hakim
> Fix For: Future
>
>
> The following unit tests are failing when building Drill with JDK 1.8.0_65:
> {noformat}
>   TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject
>   TestFrameworkTest {
>   testRepeatedColumnMatching
>   testCSVVerificationOfOrder_checkFailure
>   }
>   Drill2489CallsAfterCloseThrowExceptionsTest {
> testClosedDatabaseMetaDataMethodsThrowRight
> testClosedPlainStatementMethodsThrowRight
> testclosedPreparedStmtOfOpenConnMethodsThrowRight
> testClosedResultSetMethodsThrowRight1
> testClosedResultSetMethodsThrowRight2
>   }
>   Drill2769UnsupportedReportsUseSqlExceptionTest {
> testPreparedStatementMethodsThrowRight
> testPlainStatementMethodsThrowRight
>   }
>   TestMongoFilterPushDown {
> testFilterPushDownIsEqual
> testFilterPushDownGreaterThanWithSingleField
> testFilterPushDownLessThanWithSingleField
>   }
> TestHiveStorage {
> orderByOnHiveTable
> queryingTableWithSerDeInHiveContribJar
> queryingTablesInNonDefaultFS
> readFromAlteredPartitionedTable
> readAllSupportedHiveDataTypesNativeParquet
> queryingHiveAvroTable
> readAllSupportedHiveDataTypes
> convertFromOnHiveBinaryType
>   }
>   TestInfoSchemaOnHiveStorage {
> showDatabases
> showTablesFromDb
> defaultSchemaHive
> defaultTwoLevelSchemaHive
> describeTable1
> describeTable2
> describeTable3
> describeTable4
> describeTable5
> describeTable6
> describeTable7
> describeTable8
> describeTable9
> varCharMaxLengthAndDecimalPrecisionInInfoSchema
>   }   
>   TestSqlStdBasedAuthorization {
> showSchemas
> showTables_user0
> showTables_user1
>   }
>   TestStorageBasedHiveAuthorization {
> showSchemas
> showTablesUser0
> showTablesUser1
> showTablesUser2
>   }
>   TestViewSupportOnHiveTables {
> viewWithSelectFieldsInDef_StarInQuery
> testInfoSchemaWithHiveView
> viewWithStarInDef_SelectFieldsInQuery1
> viewWithStarInDef_SelectFieldsInQuery2
> viewWithSelectFieldsInDef_SelectFieldsInQuery
> viewWithStarInDef_StarInQuery
>   }
>   TestGeometryFunctions {
> testGeometryPointCreation
> testGeometryFromTextCreation
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)