[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeo Ogawara updated DRILL-5785:
-
Attachment: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill
sample2.pcap

Sorry, I attached html file.
Profile file is here.

You can create dummy large PCAP file with mergecap command using sample pcap 
file.

> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: 2647724c-d367-c8cc-0b6d-579228ffa31e.sys.drill, Apache 
> Drill_files.zip, Apache Drill.html, sample2.pcap
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5785) Query Error on a large PCAP file

[Robert Hou (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164029#comment-16164029
 ] 

Robert Hou commented on DRILL-5785:
---

Can you attach a sample of your PCAP file?

Can you attach the query profile?  Go to the URL http://:8047.

Click on Profiles in the upper left hand corner.  You will see a list of 
queries that have run on your system.

Click on the link for the query.  You will see something like this in the URL:

https://:8047/profiles/264e5071-f248-1001-d72a-5a4e850d6ea6

The long string is your queryID.  There should be a file 
264e5071-f248-1001-d72a-5a4e850d6ea6.sys.drill on your system in 
$DRILL_HOME/logs.  This is your query file stored on disk.  It would be helpful 
if you can attach it to this Jira.


> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: Apache Drill_files.zip, Apache Drill.html
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeo Ogawara updated DRILL-5785:
-
Attachment: Apache Drill_files.zip
Apache Drill.html

Query profiles.


> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: Apache Drill_files.zip, Apache Drill.html
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeo Ogawara updated DRILL-5785:
-
Attachment: (was: Apache Drill_files.zip)

> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: Apache Drill_files.zip, Apache Drill.html
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeo Ogawara updated DRILL-5785:
-
Attachment: (was: Apache Drill.html)

> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: Apache Drill_files.zip, Apache Drill.html
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takeo Ogawara updated DRILL-5785:
-
Attachment: Apache Drill_files.zip
Apache Drill.html

> Query Error on a large PCAP file
> 
>
> Key: DRILL-5785
> URL: https://issues.apache.org/jira/browse/DRILL-5785
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: Takeo Ogawara
>Priority: Minor
> Attachments: Apache Drill_files.zip, Apache Drill.html
>
>
> Query on a very large PCAP file (larger than 100GB) failed with following 
> error message.
> > Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
> >
> > Fragment 1:169
> >
> > [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> > (state=,code=0)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164012#comment-16164012
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138506769
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
 ---
@@ -54,11 +56,13 @@
 import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
 
 @RunWith(Parameterized.class)
+@Category({SecondaryTest.class, ParquetTest.class})
--- End diff --

Please see Terminology update in comment above.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164011#comment-16164011
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138506701
  
--- Diff: 
common/src/test/java/org/apache/drill/categories/package-info.java ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package stores the JUnit test categories.
+ */
--- End diff --

Terminology update. I'm renaming **SecondaryTest** to **SlowTest** and 
instead of **BugFixTest** I will create a category called **UnlikelyTest**. 
**UnlikelyTest** will be a category for tests that excercise specific bug 
fixes, or are for tests that you're unlikely to break.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163984#comment-16163984
 ] 

Takeo Ogawara commented on DRILL-5785:
--

Here are the messages in Drill log files.

drillbit.log
---
2017-09-11 15:06:52,390 [BitServer-2] WARN  o.a.d.exec.rpc.control.WorkEventBus 
- A fragment message arrived but there was no registered listener for that 
message: profile {
  state: FAILED
  error {
error_id: "bbf284b6-9da4-4869-ac20-fa100eed11b9"
endpoint {
  address: "node22"
  user_port: 31010
  control_port: 31011
  data_port: 31012
  version: "1.11.0"
}
error_type: SYSTEM
message: "SYSTEM ERROR: IllegalStateException: Bad magic number = 
0a0d0d0a\n\nFragment 1:200\n\n[Error Id: bbf284b6-9da4-4869-ac20-fa100eed11b9 
on node22:31010]"
exception {
  exception_class: "java.lang.IllegalStateException"
  message: "Bad magic number = 0a0d0d0a"
  stack_trace {
class_name: "com.google.common.base.Preconditions"
file_name: "Preconditions.java"
line_number: 173
method_name: "checkState"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.store.pcap.decoder.PacketDecoder"
file_name: "PacketDecoder.java"
line_number: 84
method_name: ""
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.store.pcap.PcapRecordReader"
file_name: "PcapRecordReader.java"
line_number: 104
method_name: "setup"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ScanBatch"
file_name: "ScanBatch.java"
line_number: 104
method_name: ""
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin"
file_name: "EasyFormatPlugin.java"
line_number: 166
method_name: "getReaderBatch"
is_native_method: false
  }
  stack_trace {
class_name: 
"org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator"
file_name: "EasyReaderBatchCreator.java"
line_number: 35
method_name: "getBatch"
is_native_method: false
  }
  stack_trace {
class_name: 
"org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator"
file_name: "EasyReaderBatchCreator.java"
line_number: 28
method_name: "getBatch"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 156
method_name: "getRecordBatch"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 136
method_name: "getRecordBatch"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 136
method_name: "getRecordBatch"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 179
method_name: "getChildren"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 109
method_name: "getRootExec"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.physical.impl.ImplCreator"
file_name: "ImplCreator.java"
line_number: 87
method_name: "getExec"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.exec.work.fragment.FragmentExecutor"
file_name: "FragmentExecutor.java"
line_number: 207
method_name: "run"
is_native_method: false
  }
  stack_trace {
class_name: "org.apache.drill.common.SelfCleaningRunnable"
file_name: "SelfCleaningRunnable.java"
line_number: 38
method_name: "run"
is_native_method: false
  }
  stack_trace {
class_name: "..."
line_number: 0
method_name: 

[jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163975#comment-16163975
 ] 

Timothy Farkas commented on DRILL-5783:
---

Thanks [~Paul.Rogers]. I'll look through the code and follow the same patterns. 

> Make code generation in the TopN operator more modular and test it
> --
>
> Key: DRILL-5783
> URL: https://issues.apache.org/jira/browse/DRILL-5783
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5775) Select * query on a maprdb binary table fails

[Jinfeng Ni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni reassigned DRILL-5775:
-

Assignee: Jinfeng Ni

> Select * query on a maprdb binary table fails
> -
>
> Key: DRILL-5775
> URL: https://issues.apache.org/jira/browse/DRILL-5775
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MapRDB
>Affects Versions: 1.11.0
>Reporter: Prasad Nagaraj Subramanya
>Assignee: Jinfeng Ni
>
> Select * query on a maprdb binary table fails with the below exception
> Failed with exception
> java.sql.SQLException: SYSTEM ERROR: IllegalArgumentException: 
> HBaseRecordReader does not allow column *. Column * should have been 
> converted to list of  family_n
> Commit ID - fde0a1df1734e0742b49aabdd28b02202ee2b044



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163964#comment-16163964
 ] 

Vlad Rozov commented on DRILL-5784:
---

Stack trace:
{code}
Caused by: java.lang.IndexOutOfBoundsException: index: 512, length: 4 
(expected: range(0, 512))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:122)
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:146)
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:523)
at 
org.apache.drill.exec.vector.IntVector$Accessor.get(IntVector.java:405)
at 
org.apache.drill.exec.test.generated.NestedLoopJoinGen2.doEval(NestedLoopJoinGen2.java:102)
at 
org.apache.drill.exec.physical.impl.join.NestedLoopJoinTemplate.populateOutgoingBatch(NestedLoopJoinTemplate.java:122)
at 
org.apache.drill.exec.physical.impl.join.NestedLoopJoinTemplate.outputRecords(NestedLoopJoinTemplate.java:86)
at 
org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.innerNext(NestedLoopJoinBatch.java:181)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.test.generated.HashAggregatorGen4.doWork(HashAggTemplate.java:581)
at 
org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.buildSchema(NestedLoopJoinBatch.java:380)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:141)
at 
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105)
at 
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
at 
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
... 4 common frames omitted
{code}

> SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: 
> range(0, 512))
> 
>
> 

[jira] [Commented] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163949#comment-16163949
 ] 

ASF GitHub Bot commented on DRILL-5772:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/936
  
@arina-ielchiieva, thanks for the explanation. Drill's runtime framework 
assumes that data is either:

1. ASCII (or, at least, single-byte character set based on ASCII), or
2. UTF-8 (data is converted to/from UTF-8 when converting VarChar to a Java 
String.)

Since Drill's code seems to assume ASCII (when it cares about character 
format), then one could claim that Drill does not have an encoding: it just 
treats characters as bytes. But, things such as determining string length, 
doing pattern matching, and so on must be aware of the character set -- if only 
to know which bytes are continuations of a multi-byte character. (That is, a 
three-byte sequence in UTF-8 might be one, two or three characters, depending.)

Now, if the planner assumes ISO-8859-1, but the Drill execution engine 
assumes UTF-8, then string constants passed from one to the other can become 
corrupted in the case where a particular byte sequence in ISO-8859-1 represents 
a different character than that same byte sequence in UTF-8.

Where would this occur? Look at the 
[ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) definition. ISO-8859 
is a single-byte character set with meanings associated to the bytes in the 
range 0x40 to 0x7f. But, in UTF-8, the high bit indicates a prefix character. 
So, 0xF7 is a valid single-byte character in ISO-8859, but is a lead-in 
character in UTF-8.

The point here is that setting the character set would seem to be a global 
setting. If the Saffron setting is purely for the parser (how to interpret 
incoming text), and the parser always produces Java strings in UTF-16 (which 
are then encoded into UTF-8 for execution), then we're fine.

But, if the parser encoding is written as bytes sent to the execution 
engine, we're in trouble.

Further, Drill has a web UI. The typical web character set is UTF-8, so 
queries coming from the web UI are encoded in UTF-8.

All this suggests two things:

1. Drill should either always accept UTF-8 (the Saffron property should 
always be set.) or
2. The property is specified by the client and used to decode the bytes 
within a Protobuf message to produce a character stream given to the parser.

It appears that UTF-8 is the default Protobuf String type encoding; sender 
and receiver would have to agree on another format. Does Drill have such an RPC 
property? I've not seen it, but I'm not an expert.

In short, if this change ensures that the parser *always* uses UTF-8, then 
this is good. If the character encoding is an option, then we have to consider 
all the above issues to have a fully working, end-to-end solution.


> Add unit tests to indicate how utf-8 support can be enabled / disabled in 
> Drill
> ---
>
> Key: DRILL-5772
> URL: https://issues.apache.org/jira/browse/DRILL-5772
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Add unit test to indicated how utf-8 support can be enabled in Drill.
> To select utf-8 data user needs to update system property 
> {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. 
> Calcite uses this property to get default charset, if it is not set then 
> {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite.
> This information should be also documented, probably in 
> https://drill.apache.org/docs/data-type-conversion/.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163941#comment-16163941
 ] 

ASF GitHub Bot commented on DRILL-5704:
---

Github user sohami commented on the issue:

https://github.com/apache/drill/pull/895
  
This PR was already merged by @arina-ielchiieva on 08/24


> Improve error message on client side when queries fail with "Failed to create 
> schema tree." when Impersonation is enabled and logins are anonymous
> --
>
> Key: DRILL-5704
> URL: https://issues.apache.org/jira/browse/DRILL-5704
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Reported by [~agirish]
> When username is not specified then Drill set's the session user as anonymous 
> if impersonation is enabled. During query execution Drill tries to build 
> schema tree and as part of that it validates if the user has access to the 
> workspace or not by using FileClient Api liststatus which verifies the user 
> from the OS user. Since impersonation is only enabled here without 
> authentication and we don't specify any user in connection string, Drill will 
> use default user which is "anonymous" and pass that to check workspace 
> permission which will fail as node doesn't have any valid user with that name.
> {code:java}
> Caused by: java.io.IOException: Error getting user info for current user, 
> anonymous
>..
>..
> at 
> org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> ... 10 common frames omitted
> {code}
> # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" 
> sqlline> select * from sys.drillbits;
> User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to 
> create schema tree.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163938#comment-16163938
 ] 

ASF GitHub Bot commented on DRILL-5704:
---

Github user sohami closed the pull request at:

https://github.com/apache/drill/pull/895


> Improve error message on client side when queries fail with "Failed to create 
> schema tree." when Impersonation is enabled and logins are anonymous
> --
>
> Key: DRILL-5704
> URL: https://issues.apache.org/jira/browse/DRILL-5704
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Reported by [~agirish]
> When username is not specified then Drill set's the session user as anonymous 
> if impersonation is enabled. During query execution Drill tries to build 
> schema tree and as part of that it validates if the user has access to the 
> workspace or not by using FileClient Api liststatus which verifies the user 
> from the OS user. Since impersonation is only enabled here without 
> authentication and we don't specify any user in connection string, Drill will 
> use default user which is "anonymous" and pass that to check workspace 
> permission which will fail as node doesn't have any valid user with that name.
> {code:java}
> Caused by: java.io.IOException: Error getting user info for current user, 
> anonymous
>..
>..
> at 
> org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> ... 10 common frames omitted
> {code}
> # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" 
> sqlline> select * from sys.drillbits;
> User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to 
> create schema tree.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5532) TPCH Query 16 failed when planner.width.max_per_node <10

[Jinfeng Ni (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163928#comment-16163928
 ] 

Jinfeng Ni commented on DRILL-5532:
---

This seems to be exposing a problem in execution time, after the commit 
841ead40109ff8364bee640a77881a8fea94d152 got a new query plan for the query.  

> TPCH Query 16 failed when planner.width.max_per_node <10
> 
>
> Key: DRILL-5532
> URL: https://issues.apache.org/jira/browse/DRILL-5532
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
> Environment: 10 node cluster, rhel6.4 
>Reporter: Dechang Gu
>Assignee: Jinfeng Ni
> Attachments: 
> pfofile_tpch16_gitid_8412ad4.26dce567-5ec0-8bbf-c099-f2d6307bdeef.sys.drill, 
> profile_tpch16_gitid_adbf363.26dce6f3-3d75-e906-bfcb-2c6541ee2ff0.sys.drill
>
>
> When set planner.width.max_per_node < 10 (tried 6, 8, and 9): TPCH query 16 
> failed with the following error:
> {code}
> java.sql.SQLException: SYSTEM ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
>   (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
> 
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
>   at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489)
>   at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:593)
>   at 
> org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:215)
>   at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:140)
>   at PipSQueak.fetchRows(PipSQueak.java:420)
>   at PipSQueak.runTest(PipSQueak.java:116)
>   at PipSQueak.main(PipSQueak.java:556)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM 
> ERROR: NullPointerException
> Fragment 1:1
> [Error Id: ba4661b4-2ff6-4eff-889c-fc4604a57418 on ucs-node8.perf.lab:31010]
>   (java.lang.NullPointerException) null
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe():607
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe():560
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputRecordKeysPrev():525
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.outputToBatchPrev():322
> 
> org.apache.drill.exec.test.generated.StreamingAggregatorGen250.doWork():182
> 
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():170
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
>   at 
> 

[jira] [Commented] (DRILL-5550) SELECT non-existent column produces empty required VARCHAR

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163919#comment-16163919
 ] 

ASF GitHub Bot commented on DRILL-5550:
---

Github user prasadns14 commented on the issue:

https://github.com/apache/drill/pull/939
  
@paul-rogers Can you please review the PR?


> SELECT non-existent column produces empty required VARCHAR
> --
>
> Key: DRILL-5550
> URL: https://issues.apache.org/jira/browse/DRILL-5550
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Drill's CSV column reader supports two forms of files:
> * Files with column headers as the first line of the file.
> * Files without column headers.
> The CSV storage plugin specifies which format to use for files accessed via 
> that storage plugin config.
> Suppose we have a CSV file with headers:
> {code}
> a,b,c
> 10,foo,bar
> {code}
> Suppose we configure a storage plugin to use headers:
> {code}
> TextFormatConfig csvFormat = new TextFormatConfig();
> csvFormat.fieldDelimiter = ',';
> csvFormat.skipFirstLine = false;
> csvFormat.extractHeader = true;
> {code}
> (The above can also be done using JSON when running Drill as a server.)
> Execute the following query:
> {code}
> SELECT a, c, d FROM `dfs.data.example.csv`
> {code}
> Results:
> {code}
> a,c,d
> 10,bar,
> {code}
> The actual type of column {{d}} is non-nullable VARCHAR.
> This is inconsistent with other parts of Drill in two ways, one may be a bug. 
> Most other parts of Drill use a nullable INT for "missing" columns.
> 1. For CSV it makes sense for the data type to be VARCHAR, since all CSV 
> columns are of that type.
> 2. It may *not* make sense for the column to be non-nullable and blank rather 
> than nullable and NULL. In SQL, NULL means that the data is unknown, which is 
> the case here.
> In the future, we may want to use some other indication for a missing column. 
> Until then, the requested change is to make the type of a missing CSV column 
> a nullable VARCHAR set to value NULL.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163907#comment-16163907
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138497894
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

I made ExecConstants a file class with a private constructor and added the 
static method there.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163909#comment-16163909
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138497914
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

The Properties class only allows strings to be set as values. The 
conversion to an integer is done somewhere inside the DriverManager.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))

[Vlad Rozov (JIRA)]

Vlad Rozov created DRILL-5784:
-

 Summary: SYSTEM ERROR: IndexOutOfBoundsException: index: 512, 
length: 4 (expected: range(0, 512))
 Key: DRILL-5784
 URL: https://issues.apache.org/jira/browse/DRILL-5784
 Project: Apache Drill
  Issue Type: Bug
 Environment: planner.slice_target > 10
planner.enable_nljoin_for_scalar_only = false
Reporter: Vlad Rozov
Assignee: Vlad Rozov


The following query causes IndexOutOfBoundsException:
{code}
SELECT 
  `t1`.`one` `one` 
FROM 
  (
SELECT 
  1 `one` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  INNER JOIN (
SELECT 
  314 `c_integer` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  ) `t0` ON (
`/drill/exec/java-exec/src/test/resources/join/j1`.c_integer IS NOT 
DISTINCT 
FROM 
  `t0`.`c_integer`
  ) 
GROUP BY 
  `one`
  ) `t1` 
  INNER JOIN (
SELECT 
  count(1) `measure` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  ) `t5` ON TRUE
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163892#comment-16163892
 ] 

ASF GitHub Bot commented on DRILL-5694:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495914
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
 ---
@@ -158,19 +158,17 @@ public BatchHolder(int idx) {
   } finally {
 if (!success) {
   htContainer.clear();
-  if (links != null) {
-links.clear();
-  }
+  if (links != null) { links.clear();}
 }
   }
 }
 
 private void init(IntVector links, IntVector hashValues, int size) {
   for (int i = 0; i < size; i++) {
-links.getMutator().setSafe(i, EMPTY_SLOT);
+links.getMutator().set(i, EMPTY_SLOT);
--- End diff --

This init() method is not used  looks like leftover old code 


> hash agg spill to disk, second phase OOM
> 
>
> Key: DRILL-5694
> URL: https://issues.apache.org/jira/browse/DRILL-5694
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Chun Chang
>Assignee: Boaz Ben-Zvi
>
> | 1.11.0-SNAPSHOT  | d622f76ee6336d97c9189fc589befa7b0f4189d6  | DRILL-5165: 
> For limit all case, no need to push down limit to scan  | 21.07.2017 @ 
> 10:36:29 PDT
> Second phase agg ran out of memory. Not suppose to. Test data currently only 
> accessible locally.
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
> Query:
> select row_count, sum(row_count), avg(double_field), max(double_rand), 
> count(float_rand) from parquet_500m_v1 group by row_count order by row_count 
> limit 30
> Failed with exception
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
> while executing the query.
> HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 
> 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 
> 536870912 so far allocated: 534773760.
> Fragment 1:6
> [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
>   (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 
> OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned 
> batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far 
> allocated: 534773760.
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
>   Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to 
> allocate buffer of size 

[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163887#comment-16163887
 ] 

ASF GitHub Bot commented on DRILL-5694:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495296
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java
 ---
@@ -58,7 +59,7 @@
 
   public int getHashCode(int incomingRowIdx) throws SchemaChangeException;
 
-  public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int 
hashCode) throws SchemaChangeException;
+  public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int 
hashCode) throws SchemaChangeException, RetryAfterSpillException;
--- End diff --

Done.


> hash agg spill to disk, second phase OOM
> 
>
> Key: DRILL-5694
> URL: https://issues.apache.org/jira/browse/DRILL-5694
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Chun Chang
>Assignee: Boaz Ben-Zvi
>
> | 1.11.0-SNAPSHOT  | d622f76ee6336d97c9189fc589befa7b0f4189d6  | DRILL-5165: 
> For limit all case, no need to push down limit to scan  | 21.07.2017 @ 
> 10:36:29 PDT
> Second phase agg ran out of memory. Not suppose to. Test data currently only 
> accessible locally.
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
> Query:
> select row_count, sum(row_count), avg(double_field), max(double_rand), 
> count(float_rand) from parquet_500m_v1 group by row_count order by row_count 
> limit 30
> Failed with exception
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
> while executing the query.
> HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 
> 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 
> 536870912 so far allocated: 534773760.
> Fragment 1:6
> [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
>   (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 
> OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned 
> batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far 
> allocated: 534773760.
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
>   Caused By (org.apache.drill.exec.exception.OutOfMemoryException) Unable to 
> allocate buffer of size 4194304 due to memory limit. Current allocation: 
> 534773760
> org.apache.drill.exec.memory.BaseAllocator.buffer():238
> org.apache.drill.exec.memory.BaseAllocator.buffer():213
> org.apache.drill.exec.vector.IntVector.allocateBytes():231
> 

[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163883#comment-16163883
 ] 

ASF GitHub Bot commented on DRILL-5694:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495164
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int 
incomingRowIdx) {
 hashCode >>>= bitsInMask;
 HashTable.PutStatus putStatus = null;
 long allocatedBeforeHTput = allocator.getAllocatedMemory();
-
 // ==
 // Insert the key columns into the hash table
 // ==
+boolean noReserveMem = reserveValueBatchMemory == 0;
 try {
+  if ( noReserveMem && canSpill ) { throw new 
RetryAfterSpillException();} // proactive spill, skip put()
+
   putStatus = htables[currentPartition].put(incomingRowIdx, 
htIdxHolder, hashCode);
+
+} catch (RetryAfterSpillException re) {
+  if ( ! canSpill ) { throw new 
OutOfMemoryException(getOOMErrorMsg("Can not spill")); }
--- End diff --

The method getOOMErrorMsg() does all this explanation ...


> hash agg spill to disk, second phase OOM
> 
>
> Key: DRILL-5694
> URL: https://issues.apache.org/jira/browse/DRILL-5694
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Chun Chang
>Assignee: Boaz Ben-Zvi
>
> | 1.11.0-SNAPSHOT  | d622f76ee6336d97c9189fc589befa7b0f4189d6  | DRILL-5165: 
> For limit all case, no need to push down limit to scan  | 21.07.2017 @ 
> 10:36:29 PDT
> Second phase agg ran out of memory. Not suppose to. Test data currently only 
> accessible locally.
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
> Query:
> select row_count, sum(row_count), avg(double_field), max(double_rand), 
> count(float_rand) from parquet_500m_v1 group by row_count order by row_count 
> limit 30
> Failed with exception
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
> while executing the query.
> HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 
> 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 
> 536870912 so far allocated: 534773760.
> Fragment 1:6
> [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
>   (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 
> OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned 
> batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far 
> allocated: 534773760.
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> 

[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163880#comment-16163880
 ] 

ASF GitHub Bot commented on DRILL-5694:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138494777
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int 
incomingRowIdx) {
 hashCode >>>= bitsInMask;
 HashTable.PutStatus putStatus = null;
 long allocatedBeforeHTput = allocator.getAllocatedMemory();
-
 // ==
 // Insert the key columns into the hash table
 // ==
+boolean noReserveMem = reserveValueBatchMemory == 0;
 try {
+  if ( noReserveMem && canSpill ) { throw new 
RetryAfterSpillException();} // proactive spill, skip put()
+
   putStatus = htables[currentPartition].put(incomingRowIdx, 
htIdxHolder, hashCode);
+
+} catch (RetryAfterSpillException re) {
--- End diff --

Done. 


> hash agg spill to disk, second phase OOM
> 
>
> Key: DRILL-5694
> URL: https://issues.apache.org/jira/browse/DRILL-5694
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Chun Chang
>Assignee: Boaz Ben-Zvi
>
> | 1.11.0-SNAPSHOT  | d622f76ee6336d97c9189fc589befa7b0f4189d6  | DRILL-5165: 
> For limit all case, no need to push down limit to scan  | 21.07.2017 @ 
> 10:36:29 PDT
> Second phase agg ran out of memory. Not suppose to. Test data currently only 
> accessible locally.
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
> Query:
> select row_count, sum(row_count), avg(double_field), max(double_rand), 
> count(float_rand) from parquet_500m_v1 group by row_count order by row_count 
> limit 30
> Failed with exception
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
> while executing the query.
> HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 
> 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 
> 536870912 so far allocated: 534773760.
> Fragment 1:6
> [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
>   (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 
> OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned 
> batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far 
> allocated: 534773760.
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> 
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> 

[jira] [Commented] (DRILL-5694) hash agg spill to disk, second phase OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163858#comment-16163858
 ] 

ASF GitHub Bot commented on DRILL-5694:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138492663
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -646,6 +687,46 @@ public AggOutcome doWork() {
   }
 
   /**
+   *   Use reserved values memory (if available) to try and preemp an OOM
+   */
+  private void useReservedValuesMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveValueBatchMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
+
+reserveValueBatchMemory = 0;
+  }
+  /**
+   *   Use reserved outgoing output memory (if available) to try and 
preemp an OOM
+   */
+  private void useReservedOutgoingMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveOutgoingMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
+
+reserveOutgoingMemory = 0;
+  }
+  /**
+   *  Restore the reserve memory (both)
+   *
+   */
+  private void restoreReservedMemory() {
+if ( 0 == reserveOutgoingMemory ) { // always restore OutputValues 
first (needed for spilling)
+  long memAvail = allocator.getLimit() - 
allocator.getAllocatedMemory();
+  if ( memAvail > estOutgoingAllocSize) {
+allocator.setLimit(allocator.getLimit() - estOutgoingAllocSize);
--- End diff --

Can never add twice, as the code only adds to an empty ( == 0 ) reserve.
And these are not "relative deltas", but the actual expected batch size (so 
that the following allocation would not OOM).



> hash agg spill to disk, second phase OOM
> 
>
> Key: DRILL-5694
> URL: https://issues.apache.org/jira/browse/DRILL-5694
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.11.0
>Reporter: Chun Chang
>Assignee: Boaz Ben-Zvi
>
> | 1.11.0-SNAPSHOT  | d622f76ee6336d97c9189fc589befa7b0f4189d6  | DRILL-5165: 
> For limit all case, no need to push down limit to scan  | 21.07.2017 @ 
> 10:36:29 PDT
> Second phase agg ran out of memory. Not suppose to. Test data currently only 
> accessible locally.
> /root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg15.q
> Query:
> select row_count, sum(row_count), avg(double_field), max(double_rand), 
> count(float_rand) from parquet_500m_v1 group by row_count order by row_count 
> limit 30
> Failed with exception
> java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory 
> while executing the query.
> HT was: 534773760 OOM at Second Phase. Partitions: 32. Estimated batch size: 
> 4849664. Planned batches: 0. Rows spilled so far: 6459928 Memory limit: 
> 536870912 so far allocated: 534773760.
> Fragment 1:6
> [Error Id: a193babd-f783-43da-a476-bb8dd4382420 on 10.10.30.168:31010]
>   (org.apache.drill.exec.exception.OutOfMemoryException) HT was: 534773760 
> OOM at Second Phase. Partitions: 32. Estimated batch size: 4849664. Planned 
> batches: 0. Rows spilled so far: 6459928 Memory limit: 536870912 so far 
> allocated: 534773760.
> 
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.checkGroupAndAggrValues():1175
> org.apache.drill.exec.test.generated.HashAggregatorGen1823.doWork():539
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
> org.apache.drill.exec.record.AbstractRecordBatch.next():162
>   

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163823#comment-16163823
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138489633
  
--- Diff: 
common/src/test/java/org/apache/drill/categories/package-info.java ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package stores the JUnit test categories.
+ */
--- End diff --

Currently there are two broad categories: **Fast** and **Secondary**. The 
**Fast** tests are smoke tests. The **Secondary** tests are for slow tests. I 
can add a third **BugFix**category to represent tests for specific bugs.

The rest of the categories represent broad groups of tests. Some of those 
categories have tests which span multiple class files and multiple packages. An 
example of such a category is **OperatorTest**, **PlannerTest**, or 
**SqlTest**. These categories are helpful when you are making a change in a 
particular area since they allow you to run all the tests for that area easily, 
and also effectively serve as a form of documentation. For example it would 
have been helpful to have an OptionsTest category when I was making the 
option system changes, since it took me a while to find all the relevant test 
classes and I had to construct a long maven command to run the tests.

Also the structure of the tests as they are now is somewhat counter 
intuitive because tests can span different modules. For example consider the 
vector submodule. One would think all the relevant tests for vectors are in 
that submodule, but currently they are not. In fact there are no unit tests in 
the vector submodule and all relevant vector tests are in java-exec. Also the 
memory submodule has tests in it and a relevant test in the java-exec  module 
as well.

Some categories I agree are redundant like HiveStorageTest. Those tests 
have a clear one to one mapping to their submodule, however, I included a 
category for them as well for the sake of consistency. This way we can have a 
uniform mechanism for running subsets of tests instead of one mechanism for 
some groups of tests and another mechanism for other tests.

To summarize, I can add a **BugFixTest** category to add another level of 
distinction between **Secondary** tests. But I am compelled to keep the other 
test categories intact for the reasons outlined above. Let me know what you 
think.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5704) Improve error message on client side when queries fail with "Failed to create schema tree." when Impersonation is enabled and logins are anonymous

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163749#comment-16163749
 ] 

ASF GitHub Bot commented on DRILL-5704:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/895
  
@sohami, please rebase on latest master and resolve conflicts. 


> Improve error message on client side when queries fail with "Failed to create 
> schema tree." when Impersonation is enabled and logins are anonymous
> --
>
> Key: DRILL-5704
> URL: https://issues.apache.org/jira/browse/DRILL-5704
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Reported by [~agirish]
> When username is not specified then Drill set's the session user as anonymous 
> if impersonation is enabled. During query execution Drill tries to build 
> schema tree and as part of that it validates if the user has access to the 
> workspace or not by using FileClient Api liststatus which verifies the user 
> from the OS user. Since impersonation is only enabled here without 
> authentication and we don't specify any user in connection string, Drill will 
> use default user which is "anonymous" and pass that to check workspace 
> permission which will fail as node doesn't have any valid user with that name.
> {code:java}
> Caused by: java.io.IOException: Error getting user info for current user, 
> anonymous
>..
>..
> at 
> org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:523)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:157)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.(FileSystemSchemaFactory.java:78)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:65)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:150)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:365)
>  ~[drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> at 
> org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:72)
>  [drill-java-exec-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
> ... 10 common frames omitted
> {code}
> # $DRILL_HOME/bin/sqlline -u "jdbc:drill:zk=localhost:5181" 
> sqlline> select * from sys.drillbits;
> User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Failed to 
> create schema tree.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163722#comment-16163722
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138472834
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
 ---
@@ -54,11 +56,13 @@
 import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
 
 @RunWith(Parameterized.class)
+@Category({SecondaryTest.class, ParquetTest.class})
--- End diff --

On the other hand, the Parquet writer seems pretty important and we'd want 
to test it more often.

This is why, in my earlier comment, I asked about the purpose of 
categorization: those that are worth running frequently ("smoke tests") vs. 
those that give a more thorough test, but will typically not find issues.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163725#comment-16163725
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138473108
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java
 ---
@@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() 
throws Throwable {
 
   private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws 
Throwable {
 FixtureBuilder builder = ClusterFixture.builder()
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false)
+.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
--- End diff --

These options are boot options, not runtime options, so no need for the 
prefix (unless someone changed something.)


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163723#comment-16163723
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138475004
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

I wonder, should we define a function for this pattern?

```
ExecConstants.bootDefaultFor(
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS)
```

But, we use Java 7, so the static function must reside elsewhere. Actually, 
the function might exist; I seem to recall making the same comment in the PR 
that externalized the defaults.


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163724#comment-16163724
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138472376
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java
 ---
@@ -59,6 +61,7 @@
  * Use of this option is assumed to be extremely unlikely, but it is 
included
  * for completeness.
  */
+@Category(ParquetTest.class)
--- End diff --

Secondary? Would not seem that this test need to be run on each build...


> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5752) Speed Up Unit Tests

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-5752:
--
Reviewer: Paul Rogers

> Speed Up Unit Tests
> ---
>
> Key: DRILL-5752
> URL: https://issues.apache.org/jira/browse/DRILL-5752
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Tests can be split into categories.
> High-level categories:
> * Fast
> * Slow
> Low-level categories:
> * Vector
> * WebUI
> * Planner
> * Operator
> * Storage
> * Hive
> * JDBC
> * Kudu
> * Mongo
> * Hbase
> After the tests are categorized the Travis build can just run the fast tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5752) Speed Up Unit Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163555#comment-16163555
 ] 

ASF GitHub Bot commented on DRILL-5752:
---

GitHub user ilooner opened a pull request:

https://github.com/apache/drill/pull/940

DRILL-5752 Speed Up Unit Tests add Test Categories

This PR does the following.

**Increased Concurrency:** Previously tests only executed in 2 simultaneous 
processes, now one process per core is used to execute tests. This PR also 
fixed a few issues that popped up when concurrency was increased.

**Test Categories:** Tests can now be executed according to their 
categories. There are two main categories of tests **Fast Tests** and 
**Secondary Tests**. **Fast Tests** are fast and are executed by default with 
the following command:

```
mvn -T 1C clean install
```

**Note:** -T 1C increased the number of threads used to compile classes, 
and allows us to run tests for sub modules in parallel. This command takes 
about 15 minutes to run on my laptop.

**Secondary Tests** are slower. To run both fast tests and **Secondary 
Tests** you can use the following command.

```
mvn -T 1C clean install -DexcludedGroups=""
```

In addition to **Fast** and **Secondary** tests there are more categories 
like:
- **JdbcTest**
- **ParquetTest**
- **HiveStorageTest**
- And many more

All of these categories are in the **drill-common** sub modules in 
**org.apahce.drill.categories**. All the tests are marked with one or more of 
these categories, and we can use these categories to select the tests we run. 
To run all the **Fast** tests that belong to the **JdbcTest** category we can 
use the following command:

```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" 
```

In order to run all the **Fast** AND **Secondary** JdbcTests you can use 
this command

```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" 
-DexcludedGroups=""
```



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner/drill DRILL-5752

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/940.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #940


commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65
Author: Timothy Farkas 
Date:   2017-08-30T19:53:49Z

 - Removed redundant exclusion from java-exec pom
 - Marked the slow tests in contrib as secondary tests

commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03
Author: Timothy Farkas 
Date:   2017-08-30T22:02:23Z

 - Made slow unit tests SecondaryTests
 - Changed the number of forked test processes from just 2 processes to 1.5 
process per core

commit 4159bf8f3ad4966751b9176843e240bb4885cec5
Author: Timothy Farkas 
Date:   2017-08-31T00:10:38Z

 - Added JdbcTest category

commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a
Author: Timothy Farkas 
Date:   2017-08-31T00:24:03Z

 - Removed unused imports from tests

commit bf3748c6ddd96b303762d8fde9046215cb243c02
Author: Timothy Farkas 
Date:   2017-08-31T00:34:22Z

 - Added JdbcStorageTest Category

commit 749fa9b1c4ae113a010de33bf359a38a1d56833e
Author: Timothy Farkas 
Date:   2017-08-31T18:15:18Z

 - Created the HiveStorageTest category
 - Made excludedGroups overrideable

commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042
Author: Timothy Farkas 
Date:   2017-08-31T18:39:28Z

 - Added HbaseStorageTest category

commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5
Author: Timothy Farkas 
Date:   2017-08-31T18:50:33Z

 - Added the KuduStorageTest category

commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20
Author: Timothy Farkas 
Date:   2017-08-31T19:06:34Z

 - Added the MongoStorageTest Category

commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1
Author: Timothy Farkas 
Date:   2017-08-31T19:42:09Z

 - Added a MemoryTest category

commit a0e3832511041aeb18590d65ebb7166e40be045f
Author: Timothy Farkas 
Date:   2017-08-31T19:53:42Z

 - Add another secondary test.

commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb
Author: Timothy Farkas 
Date:   2017-08-31T20:03:08Z

 - Added a SecurityTest category
 - Made more of the security tests secondary tests

commit 8dfacb0715252e063eca7d7a51895b8da191b273
Author: Timothy Farkas 
Date:   2017-08-31T20:05:05Z

 - 

[jira] [Created] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5783:
-

 Summary: Make code generation in the TopN operator more modular 
and test it
 Key: DRILL-5783
 URL: https://issues.apache.org/jira/browse/DRILL-5783
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Timothy Farkas
Assignee: Timothy Farkas






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5670) Varchar vector throws an assertion error when allocating a new vector

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163297#comment-16163297
 ] 

Paul Rogers commented on DRILL-5670:


One more thought on this one. Rahul sometimes ran these tests after disabling 
exchanges. Perhaps try that here. If the sort passes, then we can file a bug 
for the exchange issue and start focusing on that specific problem.

> Varchar vector throws an assertion error when allocating a new vector
> -
>
> Key: DRILL-5670
> URL: https://issues.apache.org/jira/browse/DRILL-5670
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Rahul Challapalli
>Assignee: Paul Rogers
> Fix For: 1.12.0
>
> Attachments: 26498995-bbad-83bc-618f-914c37a84e1f.sys.drill, 
> 26555749-4d36-10d2-6faf-e403db40c370.sys.drill, 
> 266290f3-5fdc-5873-7372-e9ee053bf867.sys.drill, 
> 269969ca-8d4d-073a-d916-9031e3d3fbf0.sys.drill, drillbit.log, drillbit.log, 
> drillbit.log, drillbit.log, drillbit.log, drillbit.log.sort, drillbit.out, 
> drill-override.conf
>
>
> I am running this test on a private branch of [paul's 
> repository|https://github.com/paul-rogers/drill]. Below is the commit info
> {code}
> git.commit.id.abbrev=d86e16c
> git.commit.user.email=prog...@maprtech.com
> git.commit.message.full=DRILL-5601\: Rollup of external sort fixes an 
> improvements\n\n- DRILL-5513\: Managed External Sort \: OOM error during the 
> merge phase\n- DRILL-5519\: Sort fails to spill and results in an OOM\n- 
> DRILL-5522\: OOM during the merge and spill process of the managed external 
> sort\n- DRILL-5594\: Excessive buffer reallocations during merge phase of 
> external sort\n- DRILL-5597\: Incorrect "bits" vector allocation in nullable 
> vectors allocateNew()\n- DRILL-5602\: Repeated List Vector fails to 
> initialize the offset vector\n\nAll of the bugs have to do with handling 
> low-memory conditions, and with\ncorrectly estimating the sizes of vectors, 
> even when those vectors come\nfrom the spill file or from an exchange. Hence, 
> the changes for all of\nthe above issues are interrelated.\n
> git.commit.id=d86e16c551e7d3553f2cde748a739b1c5a7a7659
> git.commit.message.short=DRILL-5601\: Rollup of external sort fixes an 
> improvements
> git.commit.user.name=Paul Rogers
> git.build.user.name=Rahul Challapalli
> git.commit.id.describe=0.9.0-1078-gd86e16c
> git.build.user.email=challapallira...@gmail.com
> git.branch=d86e16c551e7d3553f2cde748a739b1c5a7a7659
> git.commit.time=05.07.2017 @ 20\:34\:39 PDT
> git.build.time=12.07.2017 @ 14\:27\:03 PDT
> git.remote.origin.url=g...@github.com\:paul-rogers/drill.git
> {code}
> Below query fails with an Assertion Error
> {code}
> 0: jdbc:drill:zk=10.10.100.190:5181> ALTER SESSION SET 
> `exec.sort.disable_managed` = false;
> +---+-+
> |  ok   |   summary   |
> +---+-+
> | true  | exec.sort.disable_managed updated.  |
> +---+-+
> 1 row selected (1.044 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> alter session set 
> `planner.memory.max_query_memory_per_node` = 482344960;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | planner.memory.max_query_memory_per_node updated.  |
> +---++
> 1 row selected (0.372 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> alter session set 
> `planner.width.max_per_node` = 1;
> +---+--+
> |  ok   |   summary|
> +---+--+
> | true  | planner.width.max_per_node updated.  |
> +---+--+
> 1 row selected (0.292 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> alter session set 
> `planner.width.max_per_query` = 1;
> +---+---+
> |  ok   |summary|
> +---+---+
> | true  | planner.width.max_per_query updated.  |
> +---+---+
> 1 row selected (0.25 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> select count(*) from (select * from 
> dfs.`/drill/testdata/resource-manager/3500cols.tbl` order by 
> columns[450],columns[330],columns[230],columns[220],columns[110],columns[90],columns[80],columns[70],columns[40],columns[10],columns[20],columns[30],columns[40],columns[50],
>  
> 

[jira] [Commented] (DRILL-1162) 25 way join ended up with OOM

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163295#comment-16163295
 ] 

ASF GitHub Bot commented on DRILL-1162:
---

Github user vvysotskyi commented on the issue:

https://github.com/apache/drill/pull/905
  
@jinfengni thanks for looking into this. Completely agree with you that it 
would be better to consider both row and column count. 
Unfortunately, it does not help to fix this issue, since the ratio of 
column count of tables very often much smaller than the ratio of row count 
(actual row count). (`c1/c2 << r2/r1`)
So I guess it makes sense to implement your proposal to consider the column 
count together with the row count in another pull request.


> 25 way join ended up with OOM
> -
>
> Key: DRILL-1162
> URL: https://issues.apache.org/jira/browse/DRILL-1162
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow, Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Volodymyr Vysotskyi
>Priority: Critical
> Fix For: Future
>
> Attachments: error.log, oom_error.log
>
>
> git.commit.id.abbrev=e5c2da0
> The below query results in 0 results being returned 
> {code:sql}
> select count(*) from `lineitem1.parquet` a 
> inner join `part.parquet` j on a.l_partkey = j.p_partkey 
> inner join `orders.parquet` k on a.l_orderkey = k.o_orderkey 
> inner join `supplier.parquet` l on a.l_suppkey = l.s_suppkey 
> inner join `partsupp.parquet` m on j.p_partkey = m.ps_partkey and l.s_suppkey 
> = m.ps_suppkey 
> inner join `customer.parquet` n on k.o_custkey = n.c_custkey 
> inner join `lineitem2.parquet` b on a.l_orderkey = b.l_orderkey 
> inner join `lineitem2.parquet` c on a.l_partkey = c.l_partkey 
> inner join `lineitem2.parquet` d on a.l_suppkey = d.l_suppkey 
> inner join `lineitem2.parquet` e on a.l_extendedprice = e.l_extendedprice 
> inner join `lineitem2.parquet` f on a.l_comment = f.l_comment 
> inner join `lineitem2.parquet` g on a.l_shipdate = g.l_shipdate 
> inner join `lineitem2.parquet` h on a.l_commitdate = h.l_commitdate 
> inner join `lineitem2.parquet` i on a.l_receiptdate = i.l_receiptdate 
> inner join `lineitem2.parquet` o on a.l_receiptdate = o.l_receiptdate 
> inner join `lineitem2.parquet` p on a.l_receiptdate = p.l_receiptdate 
> inner join `lineitem2.parquet` q on a.l_receiptdate = q.l_receiptdate 
> inner join `lineitem2.parquet` r on a.l_receiptdate = r.l_receiptdate 
> inner join `lineitem2.parquet` s on a.l_receiptdate = s.l_receiptdate 
> inner join `lineitem2.parquet` t on a.l_receiptdate = t.l_receiptdate 
> inner join `lineitem2.parquet` u on a.l_receiptdate = u.l_receiptdate 
> inner join `lineitem2.parquet` v on a.l_receiptdate = v.l_receiptdate 
> inner join `lineitem2.parquet` w on a.l_receiptdate = w.l_receiptdate 
> inner join `lineitem2.parquet` x on a.l_receiptdate = x.l_receiptdate;
> {code}
> However when we remove the last 'inner join' and run the query it returns 
> '716372534'. Since the last inner join is similar to the one's before it, it 
> should match some records and return the data appropriately.
> The logs indicated that it actually returned 0 results. Attached the log file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5761) Disable Lilith ClassicMultiplexSocketAppender by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163288#comment-16163288
 ] 

ASF GitHub Bot commented on DRILL-5761:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/930#discussion_r138408349
  
--- Diff: common/src/test/resources/logback-test.xml ---
@@ -0,0 +1,92 @@
+
+
+
+
+  
+
+  
+true
+1
+true
+${LILITH_HOSTNAME:-localhost}
+  
+
+  
+
+  
+
+
+  %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
+
+  
+
+  
> Key: DRILL-5761
> URL: https://issues.apache.org/jira/browse/DRILL-5761
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> When running unit tests on the node where Hiveserver2 service is running, 
> tests run hangs in the middle. Jstack shows that some threads are waiting for 
> a condition.
> {noformat}
> Full thread dump
> "main" prio=10 tid=0x7f0998009800 nid=0x17f7 waiting on condition 
> [0x7f09a0c6d000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x00076004ebf0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>   at 
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
>   at 
> de.huxhorn.lilith.sender.MultiplexSendBytesService.sendBytes(MultiplexSendBytesService.java:132)
>   at 
> de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.sendBytes(MultiplexSocketAppenderBase.java:336)
>   at 
> de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.append(MultiplexSocketAppenderBase.java:348)
>   at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
>   at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
>   at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
>   at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
>   at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441)
>   at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395)
>   at ch.qos.logback.classic.Logger.error(Logger.java:558)
>   at 
> org.apache.drill.test.DrillTest$TestLogReporter.failed(DrillTest.java:153)
>   at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)
>   at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
>   at 

[jira] [Commented] (DRILL-5761) Disable Lilith ClassicMultiplexSocketAppender by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163284#comment-16163284
 ] 

ASF GitHub Bot commented on DRILL-5761:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/930#discussion_r138407993
  
--- Diff: common/src/test/resources/logback-test.xml ---
@@ -0,0 +1,111 @@
+
+
+
+
+  
+
+  
+true
+1
+true
+${LILITH_HOSTNAME:-localhost}
+  
+
+  
+
+  
+
+
+  %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
+
+  
+
+  
+
+  
+
--- End diff --

Can this logger be defined between line 21 and 22 in a single if condition?


> Disable Lilith ClassicMultiplexSocketAppender by default
> 
>
> Key: DRILL-5761
> URL: https://issues.apache.org/jira/browse/DRILL-5761
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> When running unit tests on the node where Hiveserver2 service is running, 
> tests run hangs in the middle. Jstack shows that some threads are waiting for 
> a condition.
> {noformat}
> Full thread dump
> "main" prio=10 tid=0x7f0998009800 nid=0x17f7 waiting on condition 
> [0x7f09a0c6d000]
>java.lang.Thread.State: WAITING (parking)
>   at sun.misc.Unsafe.park(Native Method)
>   - parking to wait for  <0x00076004ebf0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>   at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>   at 
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
>   at 
> de.huxhorn.lilith.sender.MultiplexSendBytesService.sendBytes(MultiplexSendBytesService.java:132)
>   at 
> de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.sendBytes(MultiplexSocketAppenderBase.java:336)
>   at 
> de.huxhorn.lilith.logback.appender.MultiplexSocketAppenderBase.append(MultiplexSocketAppenderBase.java:348)
>   at 
> ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
>   at 
> ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
>   at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
>   at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
>   at 
> ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441)
>   at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395)
>   at ch.qos.logback.classic.Logger.error(Logger.java:558)
>   at 
> org.apache.drill.test.DrillTest$TestLogReporter.failed(DrillTest.java:153)
>   at org.junit.rules.TestWatcher.failedQuietly(TestWatcher.java:84)
>   at org.junit.rules.TestWatcher.access$300(TestWatcher.java:46)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:62)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at org.junit.runners.Suite.runChild(Suite.java:127)
>   at org.junit.runners.Suite.runChild(Suite.java:26)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at 

[jira] [Updated] (DRILL-5782) Web UI: do not attempt to build visualized plan when plan is absent

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5782:

Summary: Web UI: do not attempt to build visualized plan when plan is 
absent  (was: Web UI: do not tattempt to build visualized plan when plan is 
absent)

> Web UI: do not attempt to build visualized plan when plan is absent
> ---
>
> Key: DRILL-5782
> URL: https://issues.apache.org/jira/browse/DRILL-5782
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
> Attachments: error_when_creating_visualized_plan.JPG
>
>
> When trying to execute query with incorrect syntax, there is an error on 
> query profile page (screenshot attached). Error occurs when we try to build 
> visualized plan when plan is absent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5782) Web UI: do not tattempt to build visualized plan when plan is absent

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5782:

Summary: Web UI: do not tattempt to build visualized plan when plan is 
absent  (was: Web UI: do not attempt build visualized plan when plan is absent)

> Web UI: do not tattempt to build visualized plan when plan is absent
> 
>
> Key: DRILL-5782
> URL: https://issues.apache.org/jira/browse/DRILL-5782
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
> Attachments: error_when_creating_visualized_plan.JPG
>
>
> When trying to execute query with incorrect syntax, there is an error on 
> query profile page (screenshot attached). Error occurs when we try to build 
> visualized plan when plan is absent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5772:

Reviewer: Paul Rogers

> Add unit tests to indicate how utf-8 support can be enabled / disabled in 
> Drill
> ---
>
> Key: DRILL-5772
> URL: https://issues.apache.org/jira/browse/DRILL-5772
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Add unit test to indicated how utf-8 support can be enabled in Drill.
> To select utf-8 data user needs to update system property 
> {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. 
> Calcite uses this property to get default charset, if it is not set then 
> {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite.
> This information should be also documented, probably in 
> https://drill.apache.org/docs/data-type-conversion/.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5772) Add unit tests to indicate how utf-8 support can be enabled / disabled in Drill

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163093#comment-16163093
 ] 

ASF GitHub Bot commented on DRILL-5772:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/936
  
@paul-rogers, the unit tests just show which influence saffron property has 
on Drill (since people in the community where asking how to enable support 
UTF-8 in Drill), a long with this PR we'll also add description to Drill 
documentation. 

So far Drill relies on Calcite saffron property to determine which charset 
it supports. By default, it's ISO-8859-1. So currently if customer wants to 
query UTF-8 data in Drill, he will get an error (one of the unit tests shows 
it) and needs to override saffron property to proceed. I guess, we don't 
support UTF-8 by default since there are some issues and Drill did not fully 
tested UTF-8 support.


> Add unit tests to indicate how utf-8 support can be enabled / disabled in 
> Drill
> ---
>
> Key: DRILL-5772
> URL: https://issues.apache.org/jira/browse/DRILL-5772
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>  Labels: doc-impacting
> Fix For: 1.12.0
>
>
> Add unit test to indicated how utf-8 support can be enabled in Drill.
> To select utf-8 data user needs to update system property 
> {{saffron.default.charset}} to {{UTF-16LE}} before starting the drillbit. 
> Calcite uses this property to get default charset, if it is not set then 
> {{ISO-8859-1}} is used by default. Drill gets default charset from Calcite.
> This information should be also documented, probably in 
> https://drill.apache.org/docs/data-type-conversion/.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-5782:
---

 Summary: Web UI: do not attempt build visualized plan when plan is 
absent
 Key: DRILL-5782
 URL: https://issues.apache.org/jira/browse/DRILL-5782
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Arina Ielchiieva
 Attachments: error_when_creating_visualized_plan.JPG

When trying to execute query with incorrect syntax, there is an error on query 
profile page (screenshot attached). Error occurs when we try to build 
visualized plan when plan is absent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5782:

Attachment: error_when_creating_visualized_plan.JPG

> Web UI: do not attempt build visualized plan when plan is absent
> 
>
> Key: DRILL-5782
> URL: https://issues.apache.org/jira/browse/DRILL-5782
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
> Attachments: error_when_creating_visualized_plan.JPG
>
>
> When trying to execute query with incorrect syntax, there is an error on 
> query profile page (screenshot attached). Error occurs when we try to build 
> visualized plan when plan is absent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5338.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query 
> Profile that they are part of total time duration
> ---
>
> Key: DRILL-5338
> URL: https://issues.apache.org/jira/browse/DRILL-5338
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.12.0
>
> Attachments: QueryProfile.JPG
>
>
> Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile 
> that they are part of total time duration
> Screenshot is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5339) Web UI: flaws in query profile display on error

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5339.
-
Resolution: Fixed

Points 1-4 fixed in the scope of DRILL-5766.

> Web UI: flaws in query profile display on error
> ---
>
> Key: DRILL-5339
> URL: https://issues.apache.org/jira/browse/DRILL-5339
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: error_shift.JPG, fragment_profiles.JPG, 
> verbose_error.JPG, visualized_plan.JPG
>
>
> 1. Error is shifted to the right (screenshot - error_shift.jpg)
> 2. When Fragment Profiles doesn't have any data table, there is offset in 
> overview tab (screenshot - fragment_profiles.jpg)
> 3. Verbose error message tab can be designed like overview tab to be more 
> concise (screenshot - verbose_error.jpg)
> 4. Remove three dots in the end of "Verbose error message..." .
> 5. When visualized plan is empty, there is big offset (screenshot - 
> visualized_plan.jpg)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5339) Web UI: flaws in query profile display on error

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5339:
---

Assignee: Arina Ielchiieva

> Web UI: flaws in query profile display on error
> ---
>
> Key: DRILL-5339
> URL: https://issues.apache.org/jira/browse/DRILL-5339
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: error_shift.JPG, fragment_profiles.JPG, 
> verbose_error.JPG, visualized_plan.JPG
>
>
> 1. Error is shifted to the right (screenshot - error_shift.jpg)
> 2. When Fragment Profiles doesn't have any data table, there is offset in 
> overview tab (screenshot - fragment_profiles.jpg)
> 3. Verbose error message tab can be designed like overview tab to be more 
> concise (screenshot - verbose_error.jpg)
> 4. Remove three dots in the end of "Verbose error message..." .
> 5. When visualized plan is empty, there is big offset (screenshot - 
> visualized_plan.jpg)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5338:
---

Assignee: Arina Ielchiieva

> Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query 
> Profile that they are part of total time duration
> ---
>
> Key: DRILL-5338
> URL: https://issues.apache.org/jira/browse/DRILL-5338
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.12.0
>
> Attachments: QueryProfile.JPG
>
>
> Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile 
> that they are part of total time duration
> Screenshot is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5338:

Fix Version/s: 1.12.0

> Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query 
> Profile that they are part of total time duration
> ---
>
> Key: DRILL-5338
> URL: https://issues.apache.org/jira/browse/DRILL-5338
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
> Fix For: 1.12.0
>
> Attachments: QueryProfile.JPG
>
>
> Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile 
> that they are part of total time duration
> Screenshot is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5341.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: remove duplicating link to documentation in Options page
> 
>
> Key: DRILL-5341
> URL: https://issues.apache.org/jira/browse/DRILL-5341
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: doc_link.JPG
>
>
> Remove link to Documentation on Options page (http://localhost:8047/options) 
> as we have the same link on page header. Screenshot - doc_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5346) Web UI: remove link on user in query profile list

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5346:
---

Assignee: Arina Ielchiieva

> Web UI: remove link on user in query profile list
> -
>
> Key: DRILL-5346
> URL: https://issues.apache.org/jira/browse/DRILL-5346
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: query_profile_link.JPG
>
>
> Remove link on user column that leads to query profile. Since we don't have 
> user page information, it's better to leave user name without link to avoid 
> confusion. Plus we already have link that leads to query profile on query 
> itself.
> Page - http://localhost:8047/profiles
> Screenshot - query_profile_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5346) Web UI: remove link on user in query profile list

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5346.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: remove link on user in query profile list
> -
>
> Key: DRILL-5346
> URL: https://issues.apache.org/jira/browse/DRILL-5346
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: query_profile_link.JPG
>
>
> Remove link on user column that leads to query profile. Since we don't have 
> user page information, it's better to leave user name without link to avoid 
> confusion. Plus we already have link that leads to query profile on query 
> itself.
> Page - http://localhost:8047/profiles
> Screenshot - query_profile_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-5341:
---

Assignee: Arina Ielchiieva

> Web UI: remove duplicating link to documentation in Options page
> 
>
> Key: DRILL-5341
> URL: https://issues.apache.org/jira/browse/DRILL-5341
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: doc_link.JPG
>
>
> Remove link to Documentation on Options page (http://localhost:8047/options) 
> as we have the same link on page header. Screenshot - doc_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5346) Web UI: remove link on user in query profile list

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5346:

Fix Version/s: 1.12.0

> Web UI: remove link on user in query profile list
> -
>
> Key: DRILL-5346
> URL: https://issues.apache.org/jira/browse/DRILL-5346
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: query_profile_link.JPG
>
>
> Remove link on user column that leads to query profile. Since we don't have 
> user page information, it's better to leave user name without link to avoid 
> confusion. Plus we already have link that leads to query profile on query 
> itself.
> Page - http://localhost:8047/profiles
> Screenshot - query_profile_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5377) Five-digit year dates are displayed incorrectly via jdbc

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5377:

Reviewer: Paul Rogers

> Five-digit year dates are displayed incorrectly via jdbc
> 
>
> Key: DRILL-5377
> URL: https://issues.apache.org/jira/browse/DRILL-5377
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.10.0
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
> Fix For: 1.12.0
>
>
> git.commit.id.abbrev=38ef562
> The issue is connected to displaying five-digit year dates via jdbc
> Below is the output, I get from test framework when I disable auto correction 
> for date fields
> {code}
> select l_shipdate from table(cp.`tpch/lineitem.parquet` (type => 'parquet', 
> autoCorrectCorruptDates => false)) order by l_shipdate limit 10;
> ^@356-03-19
> ^@356-03-21
> ^@356-03-21
> ^@356-03-23
> ^@356-03-24
> ^@356-03-24
> ^@356-03-26
> ^@356-03-26
> ^@356-03-26
> ^@356-03-26
> {code}
> Or a simpler case:
> {code}
> 0: jdbc:drill:> select cast('11356-02-16' as date) as FUTURE_DATE from 
> (VALUES(1));
> +--+
> | FUTURE_DATE  |
> +--+
> | 356-02-16   |
> +--+
> 1 row selected (0.293 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5377) Five-digit year dates are displayed incorrectly via jdbc

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5377:

Labels:   (was: ready-to-commit)

> Five-digit year dates are displayed incorrectly via jdbc
> 
>
> Key: DRILL-5377
> URL: https://issues.apache.org/jira/browse/DRILL-5377
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.10.0
>Reporter: Rahul Challapalli
>Assignee: Vitalii Diravka
> Fix For: 1.12.0
>
>
> git.commit.id.abbrev=38ef562
> The issue is connected to displaying five-digit year dates via jdbc
> Below is the output, I get from test framework when I disable auto correction 
> for date fields
> {code}
> select l_shipdate from table(cp.`tpch/lineitem.parquet` (type => 'parquet', 
> autoCorrectCorruptDates => false)) order by l_shipdate limit 10;
> ^@356-03-19
> ^@356-03-21
> ^@356-03-21
> ^@356-03-23
> ^@356-03-24
> ^@356-03-24
> ^@356-03-26
> ^@356-03-26
> ^@356-03-26
> ^@356-03-26
> {code}
> Or a simpler case:
> {code}
> 0: jdbc:drill:> select cast('11356-02-16' as date) as FUTURE_DATE from 
> (VALUES(1));
> +--+
> | FUTURE_DATE  |
> +--+
> | 356-02-16   |
> +--+
> 1 row selected (0.293 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5781) Fix unit test failures to use tests config even if default config is available

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5781:

Issue Type: Task  (was: Bug)

> Fix unit test failures to use tests config even if default config is available
> --
>
> Key: DRILL-5781
> URL: https://issues.apache.org/jira/browse/DRILL-5781
> Project: Apache Drill
>  Issue Type: Task
>Affects Versions: 1.11.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>
> Unit tests fail when they are run with the mapr profile.
> Tests failures, connected with the Zookeeper configuration that differs from 
> expected:
> {noformat}
> DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
> Coul...
>   TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure 
> server...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists 
> » IO
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not 
> conf...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure 
> serve...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not 
> configure...
>   TestEphemeralStore.tearDown:132 NullPointer
>   TestZookeeperClient.testGetWithVersion » IO Could not configure server 
> because...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO 
> Could n...
>   TestEphemeralStore.tearDown:132 NullPointer
>   TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure 
> ser...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not 
> con...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure 
> server...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure 
> server...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testPutAndGetWorks » IO Could not configure server 
> because...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure 
> server ...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could 
> not...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testCreate » IO Could not configure server because SASL 
> co...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testDelete » IO Could not configure server because SASL 
> co...
>   TestZookeeperClient.tearDown:86 NullPointer
>   TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not 
> configure s...
>   TestZookeeperClient.tearDown:86 NullPointer
> TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
> ...
>   TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: 
> Could...
>   TestExceptionInjection.injectionOnSpecificBit:217 » Runtime 
> java.io.IOExceptio...
> HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named 
> 'Serv...
> {noformat}
> Test failures, connected with Hadoop configuration that differs from expected:
> {noformat}
> TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
>  » ClassCast
>   
> TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
>  » ClassCast
>   
> TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106
>  » Runtime
>   
> TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
>  » ClassCast
> TestHiveStorage>HiveTestBase.generateHive:34 » Runtime 
> java.lang.RuntimeExcept...
>   TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime 
> java.lang.R...
>   TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
> sett...
>   TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
> setti...
>   
> TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
>  » ClassCast
>   
> TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
>  » ClassCast
>   

[jira] [Created] (DRILL-5781) Fix unit test failures to use tests config even if default config is available

[Volodymyr Vysotskyi (JIRA)]

Volodymyr Vysotskyi created DRILL-5781:
--

 Summary: Fix unit test failures to use tests config even if 
default config is available
 Key: DRILL-5781
 URL: https://issues.apache.org/jira/browse/DRILL-5781
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi


Unit tests fail when they are run with the mapr profile.
Tests failures, connected with the Zookeeper configuration that differs from 
expected:
{noformat}
DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
Coul...
  TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists » 
IO
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not 
conf...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure 
serve...
  TestZookeeperClient.tearDown:86 NullPointer
  TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not 
configure...
  TestEphemeralStore.tearDown:132 NullPointer
  TestZookeeperClient.testGetWithVersion » IO Could not configure server 
because...
  TestZookeeperClient.tearDown:86 NullPointer
  TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO Could 
n...
  TestEphemeralStore.tearDown:132 NullPointer
  TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure 
ser...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not 
con...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutAndGetWorks » IO Could not configure server 
because...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure server 
...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could 
not...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testCreate » IO Could not configure server because SASL 
co...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testDelete » IO Could not configure server because SASL 
co...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not configure 
s...
  TestZookeeperClient.tearDown:86 NullPointer
TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
...
  TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: 
Could...
  TestExceptionInjection.injectionOnSpecificBit:217 » Runtime 
java.io.IOExceptio...

HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named 
'Serv...
{noformat}

Test failures, connected with Hadoop configuration that differs from expected:
{noformat}
TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106
 » Runtime
  
TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast

TestHiveStorage>HiveTestBase.generateHive:34 » Runtime 
java.lang.RuntimeExcept...
  TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime 
java.lang.R...
  TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
sett...
  TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
setti...
  
TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  TestHivePartitionPruning>HiveTestBase.generateHive:35 » ExecutionSetup 
Failure...
  TestViewSupportOnHiveTables.generateHive:35 » ExecutionSetup Failure setting 
u...
  TestHiveProjectPushDown>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
...
{noformat}

Test specific configuration should be added to fix these failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)