[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138506769
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
 ---
@@ -54,11 +56,13 @@
 import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
 
 @RunWith(Parameterized.class)
+@Category({SecondaryTest.class, ParquetTest.class})
--- End diff --

Please see Terminology update in comment above.


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138506701
  
--- Diff: 
common/src/test/java/org/apache/drill/categories/package-info.java ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package stores the JUnit test categories.
+ */
--- End diff --

Terminology update. I'm renaming **SecondaryTest** to **SlowTest** and 
instead of **BugFixTest** I will create a category called **UnlikelyTest**. 
**UnlikelyTest** will be a category for tests that excercise specific bug 
fixes, or are for tests that you're unlikely to break.


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138505403
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java
 ---
@@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() 
throws Throwable {
 
   private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws 
Throwable {
 FixtureBuilder builder = ClusterFixture.builder()
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false)
+.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
--- End diff --

Done, thanks for catching.


---

[jira] [Created] (DRILL-5785) Query Error on a large PCAP file

[Takeo Ogawara (JIRA)]

Takeo Ogawara created DRILL-5785:


 Summary: Query Error on a large PCAP file
 Key: DRILL-5785
 URL: https://issues.apache.org/jira/browse/DRILL-5785
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Other
Affects Versions: 1.11.0
Reporter: Takeo Ogawara
Priority: Minor


Query on a very large PCAP file (larger than 100GB) failed with following error 
message.
> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
>
> Fragment 1:169
>
> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010] 
> (state=,code=0)





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill issue #936: DRILL-5772: Add unit tests to indicate how utf-8 support c...

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/936
  
@arina-ielchiieva, thanks for the explanation. Drill's runtime framework 
assumes that data is either:

1. ASCII (or, at least, single-byte character set based on ASCII), or
2. UTF-8 (data is converted to/from UTF-8 when converting VarChar to a Java 
String.)

Since Drill's code seems to assume ASCII (when it cares about character 
format), then one could claim that Drill does not have an encoding: it just 
treats characters as bytes. But, things such as determining string length, 
doing pattern matching, and so on must be aware of the character set -- if only 
to know which bytes are continuations of a multi-byte character. (That is, a 
three-byte sequence in UTF-8 might be one, two or three characters, depending.)

Now, if the planner assumes ISO-8859-1, but the Drill execution engine 
assumes UTF-8, then string constants passed from one to the other can become 
corrupted in the case where a particular byte sequence in ISO-8859-1 represents 
a different character than that same byte sequence in UTF-8.

Where would this occur? Look at the 
[ISO-8859-1](https://en.wikipedia.org/wiki/ISO/IEC_8859-1) definition. ISO-8859 
is a single-byte character set with meanings associated to the bytes in the 
range 0x40 to 0x7f. But, in UTF-8, the high bit indicates a prefix character. 
So, 0xF7 is a valid single-byte character in ISO-8859, but is a lead-in 
character in UTF-8.

The point here is that setting the character set would seem to be a global 
setting. If the Saffron setting is purely for the parser (how to interpret 
incoming text), and the parser always produces Java strings in UTF-16 (which 
are then encoded into UTF-8 for execution), then we're fine.

But, if the parser encoding is written as bytes sent to the execution 
engine, we're in trouble.

Further, Drill has a web UI. The typical web character set is UTF-8, so 
queries coming from the web UI are encoded in UTF-8.

All this suggests two things:

1. Drill should either always accept UTF-8 (the Saffron property should 
always be set.) or
2. The property is specified by the client and used to decode the bytes 
within a Protobuf message to produce a character stream given to the parser.

It appears that UTF-8 is the default Protobuf String type encoding; sender 
and receiver would have to agree on another format. Does Drill have such an RPC 
property? I've not seen it, but I'm not an expert.

In short, if this change ensures that the parser *always* uses UTF-8, then 
this is good. If the character encoding is an option, then we have to consider 
all the above issues to have a fully working, end-to-end solution.


---

[GitHub] drill issue #895: DRILL-5704: Improve error message on client side when quer...

[sohami]

Github user sohami commented on the issue:

https://github.com/apache/drill/pull/895
  
This PR was already merged by @arina-ielchiieva on 08/24


---

[GitHub] drill pull request #895: DRILL-5704: Improve error message on client side wh...

[sohami]

Github user sohami closed the pull request at:

https://github.com/apache/drill/pull/895


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138499857
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
 ---
@@ -54,11 +56,13 @@
 import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
 
 @RunWith(Parameterized.class)
+@Category({SecondaryTest.class, ParquetTest.class})
--- End diff --

To me the main distinction between smoke and secondary tests should be 
runtime. It's hard in my mind to distinguish based on importance, because the 
relevance of a test depends on the change being done. And you would end up 
running all the tests before finishing your change no matter what. The workflow 
I had in mind was the following:

  1. Make a change
  1. Run Fast tests for my category of interest (e.g. **OperatorTest**). 
Oops there's a failure
  1. Fix the failure
  1. Run Fast tests for category **OperatorTest** again. Oops another 
failure
  1. Fix the failure
  1. Run Fast tests for category **OperatorTest** again. Yay they pass
  1. Run all the tests
  1. Done

Let me know what you think.


---

[GitHub] drill issue #939: DRILL-5550: Missing CSV column value set to null

[prasadns14]

Github user prasadns14 commented on the issue:

https://github.com/apache/drill/pull/939
  
@paul-rogers Can you please review the PR?


---

[jira] [Created] (DRILL-5784) SYSTEM ERROR: IndexOutOfBoundsException: index: 512, length: 4 (expected: range(0, 512))

[Vlad Rozov (JIRA)]

Vlad Rozov created DRILL-5784:
-

 Summary: SYSTEM ERROR: IndexOutOfBoundsException: index: 512, 
length: 4 (expected: range(0, 512))
 Key: DRILL-5784
 URL: https://issues.apache.org/jira/browse/DRILL-5784
 Project: Apache Drill
  Issue Type: Bug
 Environment: planner.slice_target > 10
planner.enable_nljoin_for_scalar_only = false
Reporter: Vlad Rozov
Assignee: Vlad Rozov


The following query causes IndexOutOfBoundsException:
{code}
SELECT 
  `t1`.`one` `one` 
FROM 
  (
SELECT 
  1 `one` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  INNER JOIN (
SELECT 
  314 `c_integer` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  ) `t0` ON (
`/drill/exec/java-exec/src/test/resources/join/j1`.c_integer IS NOT 
DISTINCT 
FROM 
  `t0`.`c_integer`
  ) 
GROUP BY 
  `one`
  ) `t1` 
  INNER JOIN (
SELECT 
  count(1) `measure` 
FROM 
  dfs.`/drill/exec/java-exec/src/test/resources/join/j1`
  ) `t5` ON TRUE
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138497914
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

The Properties class only allows strings to be set as values. The 
conversion to an integer is done somewhere inside the DriverManager.


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138497894
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

I made ExecConstants a file class with a private constructor and added the 
static method there.


---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495914
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
 ---
@@ -158,19 +158,17 @@ public BatchHolder(int idx) {
   } finally {
 if (!success) {
   htContainer.clear();
-  if (links != null) {
-links.clear();
-  }
+  if (links != null) { links.clear();}
 }
   }
 }
 
 private void init(IntVector links, IntVector hashValues, int size) {
   for (int i = 0; i < size; i++) {
-links.getMutator().setSafe(i, EMPTY_SLOT);
+links.getMutator().set(i, EMPTY_SLOT);
--- End diff --

This init() method is not used  looks like leftover old code 


---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495296
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java
 ---
@@ -58,7 +59,7 @@
 
   public int getHashCode(int incomingRowIdx) throws SchemaChangeException;
 
-  public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int 
hashCode) throws SchemaChangeException;
+  public PutStatus put(int incomingRowIdx, IndexPointer htIdxHolder, int 
hashCode) throws SchemaChangeException, RetryAfterSpillException;
--- End diff --

Done.


---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138495164
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int 
incomingRowIdx) {
 hashCode >>>= bitsInMask;
 HashTable.PutStatus putStatus = null;
 long allocatedBeforeHTput = allocator.getAllocatedMemory();
-
 // ==
 // Insert the key columns into the hash table
 // ==
+boolean noReserveMem = reserveValueBatchMemory == 0;
 try {
+  if ( noReserveMem && canSpill ) { throw new 
RetryAfterSpillException();} // proactive spill, skip put()
+
   putStatus = htables[currentPartition].put(incomingRowIdx, 
htIdxHolder, hashCode);
+
+} catch (RetryAfterSpillException re) {
+  if ( ! canSpill ) { throw new 
OutOfMemoryException(getOOMErrorMsg("Can not spill")); }
--- End diff --

The method getOOMErrorMsg() does all this explanation ...


---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138494777
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -1178,20 +1273,38 @@ private void checkGroupAndAggrValues(int 
incomingRowIdx) {
 hashCode >>>= bitsInMask;
 HashTable.PutStatus putStatus = null;
 long allocatedBeforeHTput = allocator.getAllocatedMemory();
-
 // ==
 // Insert the key columns into the hash table
 // ==
+boolean noReserveMem = reserveValueBatchMemory == 0;
 try {
+  if ( noReserveMem && canSpill ) { throw new 
RetryAfterSpillException();} // proactive spill, skip put()
+
   putStatus = htables[currentPartition].put(incomingRowIdx, 
htIdxHolder, hashCode);
+
+} catch (RetryAfterSpillException re) {
--- End diff --

Done. 


---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138492663
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -646,6 +687,46 @@ public AggOutcome doWork() {
   }
 
   /**
+   *   Use reserved values memory (if available) to try and preemp an OOM
+   */
+  private void useReservedValuesMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveValueBatchMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
+
+reserveValueBatchMemory = 0;
+  }
+  /**
+   *   Use reserved outgoing output memory (if available) to try and 
preemp an OOM
+   */
+  private void useReservedOutgoingMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveOutgoingMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
+
+reserveOutgoingMemory = 0;
+  }
+  /**
+   *  Restore the reserve memory (both)
+   *
+   */
+  private void restoreReservedMemory() {
+if ( 0 == reserveOutgoingMemory ) { // always restore OutputValues 
first (needed for spilling)
+  long memAvail = allocator.getLimit() - 
allocator.getAllocatedMemory();
+  if ( memAvail > estOutgoingAllocSize) {
+allocator.setLimit(allocator.getLimit() - estOutgoingAllocSize);
--- End diff --

Can never add twice, as the code only adds to an empty ( == 0 ) reserve.
And these are not "relative deltas", but the actual expected batch size (so 
that the following allocation would not OOM).



---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138489633
  
--- Diff: 
common/src/test/java/org/apache/drill/categories/package-info.java ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package stores the JUnit test categories.
+ */
--- End diff --

Currently there are two broad categories: **Fast** and **Secondary**. The 
**Fast** tests are smoke tests. The **Secondary** tests are for slow tests. I 
can add a third **BugFix**category to represent tests for specific bugs.

The rest of the categories represent broad groups of tests. Some of those 
categories have tests which span multiple class files and multiple packages. An 
example of such a category is **OperatorTest**, **PlannerTest**, or 
**SqlTest**. These categories are helpful when you are making a change in a 
particular area since they allow you to run all the tests for that area easily, 
and also effectively serve as a form of documentation. For example it would 
have been helpful to have an OptionsTest category when I was making the 
option system changes, since it took me a while to find all the relevant test 
classes and I had to construct a long maven command to run the tests.

Also the structure of the tests as they are now is somewhat counter 
intuitive because tests can span different modules. For example consider the 
vector submodule. One would think all the relevant tests for vectors are in 
that submodule, but currently they are not. In fact there are no unit tests in 
the vector submodule and all relevant vector tests are in java-exec. Also the 
memory submodule has tests in it and a relevant test in the java-exec  module 
as well.

Some categories I agree are redundant like HiveStorageTest. Those tests 
have a clear one to one mapping to their submodule, however, I included a 
category for them as well for the sake of consistency. This way we can have a 
uniform mechanism for running subsets of tests instead of one mechanism for 
some groups of tests and another mechanism for other tests.

To summarize, I can add a **BugFixTest** category to add another level of 
distinction between **Secondary** tests. But I am compelled to keep the other 
test categories intact for the reasons outlined above. Let me know what you 
think.


---

[GitHub] drill issue #895: DRILL-5704: Improve error message on client side when quer...

[paul-rogers]

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/895
  
@sohami, please rebase on latest master and resolve conflicts. 


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138473108
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/xsort/TestSimpleExternalSort.java
 ---
@@ -148,10 +149,10 @@ public void sortOneKeyDescendingExternalSortLegacy() 
throws Throwable {
 
   private void sortOneKeyDescendingExternalSort(boolean testLegacy) throws 
Throwable {
 FixtureBuilder builder = ClusterFixture.builder()
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_SPILL_GROUP_SIZE, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_BATCH_LIMIT, 4)
-.configProperty(ExecConstants.EXTERNAL_SORT_DISABLE_MANAGED, false)
+.configProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.EXTERNAL_SORT_SPILL_THRESHOLD, 4)
--- End diff --

These options are boot options, not runtime options, so no need for the 
prefix (unless someone changed something.)


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138472376
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestCorruptParquetDateCorrection.java
 ---
@@ -59,6 +61,7 @@
  * Use of this option is assumed to be extremely unlikely, but it is 
included
  * for completeness.
  */
+@Category(ParquetTest.class)
--- End diff --

Secondary? Would not seem that this test need to be run on each build...


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138472834
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java
 ---
@@ -54,11 +56,13 @@
 import org.junit.Ignore;
 import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.rules.TemporaryFolder;
 import org.junit.runner.RunWith;
 import org.junit.runners.Parameterized;
 
 @RunWith(Parameterized.class)
+@Category({SecondaryTest.class, ParquetTest.class})
--- End diff --

On the other hand, the Parquet writer seems pretty important and we'd want 
to test it more often.

This is why, in my earlier comment, I asked about the purpose of 
categorization: those that are worth running frequently ("smoke tests") vs. 
those that give a more thorough test, but will typically not find issues.


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138475152
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

Also, since properties don't have to be strings, why is this property 
stored as a string when the value is an int?


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138466695
  
--- Diff: 
common/src/test/java/org/apache/drill/categories/package-info.java ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package stores the JUnit test categories.
+ */
--- End diff --

Very cool. One thing that is not intuitively obvious here is the purpose 
and usage of the categories.

On the one hand, we already have modules for memory, vectors, Hive, etc. 
One can easily run just those tests by cd'ing into that module and running 
Maven from there.

So, one would imagine categories to provide another dimension, orthogonal 
to modules. For example, the classic "fast" vs. "slow." One could also imagine 
a core set of "smoke" tests that run quickly and do a quick validation. Then a 
set of standard tests that are more thorough. Then, very in-depth tests that 
either take a long time, or validate very specific bugs or regressions.

Can you add a bit of commentary to advise how you envision the categories 
to be used?


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[paul-rogers]

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/940#discussion_r138475004
  
--- Diff: exec/jdbc/src/test/java/org/apache/drill/jdbc/ConnectionTest.java 
---
@@ -57,7 +61,9 @@ public static void setUpConnection() throws SQLException {
 // Connection--and other JDBC objects--on test method failure, but 
this test
 // class uses some objects across methods.)
 Driver.load();
-connection = DriverManager.getConnection( "jdbc:drill:zk=local" );
+Properties properties = new Properties();
+properties.setProperty(OptionValidator.OPTION_DEFAULTS_ROOT + 
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS, "3");
--- End diff --

I wonder, should we define a function for this pattern?

```
ExecConstants.bootDefaultFor(
ExecConstants.CREATE_PREPARE_STATEMENT_TIMEOUT_MILLIS)
```

But, we use Java 7, so the static function must reside elsewhere. Actually, 
the function might exist; I seem to recall making the same comment in the PR 
that externalized the defaults.


---

[GitHub] drill issue #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

Github user ilooner commented on the issue:

https://github.com/apache/drill/pull/940
  
@paul-rogers This is an implementation of the Test Categories suggestion 
you made on the dev list. It's ready for review.


---

[GitHub] drill pull request #940: DRILL-5752 Speed Up Unit Tests add Test Categories

[ilooner]

GitHub user ilooner opened a pull request:

https://github.com/apache/drill/pull/940

DRILL-5752 Speed Up Unit Tests add Test Categories

This PR does the following.

**Increased Concurrency:** Previously tests only executed in 2 simultaneous 
processes, now one process per core is used to execute tests. This PR also 
fixed a few issues that popped up when concurrency was increased.

**Test Categories:** Tests can now be executed according to their 
categories. There are two main categories of tests **Fast Tests** and 
**Secondary Tests**. **Fast Tests** are fast and are executed by default with 
the following command:

```
mvn -T 1C clean install
```

**Note:** -T 1C increased the number of threads used to compile classes, 
and allows us to run tests for sub modules in parallel. This command takes 
about 15 minutes to run on my laptop.

**Secondary Tests** are slower. To run both fast tests and **Secondary 
Tests** you can use the following command.

```
mvn -T 1C clean install -DexcludedGroups=""
```

In addition to **Fast** and **Secondary** tests there are more categories 
like:
- **JdbcTest**
- **ParquetTest**
- **HiveStorageTest**
- And many more

All of these categories are in the **drill-common** sub modules in 
**org.apahce.drill.categories**. All the tests are marked with one or more of 
these categories, and we can use these categories to select the tests we run. 
To run all the **Fast** tests that belong to the **JdbcTest** category we can 
use the following command:

```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" 
```

In order to run all the **Fast** AND **Secondary** JdbcTests you can use 
this command

```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest" 
-DexcludedGroups=""
```



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ilooner/drill DRILL-5752

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/940.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #940


commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65
Author: Timothy Farkas 
Date:   2017-08-30T19:53:49Z

 - Removed redundant exclusion from java-exec pom
 - Marked the slow tests in contrib as secondary tests

commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03
Author: Timothy Farkas 
Date:   2017-08-30T22:02:23Z

 - Made slow unit tests SecondaryTests
 - Changed the number of forked test processes from just 2 processes to 1.5 
process per core

commit 4159bf8f3ad4966751b9176843e240bb4885cec5
Author: Timothy Farkas 
Date:   2017-08-31T00:10:38Z

 - Added JdbcTest category

commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a
Author: Timothy Farkas 
Date:   2017-08-31T00:24:03Z

 - Removed unused imports from tests

commit bf3748c6ddd96b303762d8fde9046215cb243c02
Author: Timothy Farkas 
Date:   2017-08-31T00:34:22Z

 - Added JdbcStorageTest Category

commit 749fa9b1c4ae113a010de33bf359a38a1d56833e
Author: Timothy Farkas 
Date:   2017-08-31T18:15:18Z

 - Created the HiveStorageTest category
 - Made excludedGroups overrideable

commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042
Author: Timothy Farkas 
Date:   2017-08-31T18:39:28Z

 - Added HbaseStorageTest category

commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5
Author: Timothy Farkas 
Date:   2017-08-31T18:50:33Z

 - Added the KuduStorageTest category

commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20
Author: Timothy Farkas 
Date:   2017-08-31T19:06:34Z

 - Added the MongoStorageTest Category

commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1
Author: Timothy Farkas 
Date:   2017-08-31T19:42:09Z

 - Added a MemoryTest category

commit a0e3832511041aeb18590d65ebb7166e40be045f
Author: Timothy Farkas 
Date:   2017-08-31T19:53:42Z

 - Add another secondary test.

commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb
Author: Timothy Farkas 
Date:   2017-08-31T20:03:08Z

 - Added a SecurityTest category
 - Made more of the security tests secondary tests

commit 8dfacb0715252e063eca7d7a51895b8da191b273
Author: Timothy Farkas 
Date:   2017-08-31T20:05:05Z

 - Removed unused import for security test

commit 72c3db7f1774fb0e37dd61a25ef1215f0391da63
Author: Timothy Farkas 
Date:   2017-08-31T20:05:57Z

 - Added another memory test

commit 

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138437442
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -646,6 +687,46 @@ public AggOutcome doWork() {
   }
 
   /**
+   *   Use reserved values memory (if available) to try and preemp an OOM
+   */
+  private void useReservedValuesMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveValueBatchMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
+
+reserveValueBatchMemory = 0;
+  }
+  /**
+   *   Use reserved outgoing output memory (if available) to try and 
preemp an OOM
+   */
+  private void useReservedOutgoingMemory() {
+// try to preempt an OOM by using the reserved memory
+long reservedMemory = reserveOutgoingMemory;
+if ( reservedMemory > 0 ) { allocator.setLimit(allocator.getLimit() + 
reservedMemory); }
--- End diff --

   Because the first uncontrolled memory allocation happens when inserting 
into the hash table (i.e. put()).  Given this "uncontrollability", better OOM 
there (which we can handle, by spilling and retrying). Now if all the memory 
was "given" in the limit, the put() may not OOM, but leave too little available 
memory to continue (i.e. to create a values batch, or an outgoing batch) -- 
these situations we can not handle.
   By subtracting from the limit a "reserve" for these two batches, we may 
force a put() OOM early (but that's OK). But we also ensure that the following 
two batches could be allocated. In some way this is similar to having multiple 
dedicated allocators, only simpler.
   Adding or subtracting is just an operation on a local field; no 
performance effect in any way. Also by using a single allocator we can handle 
cases like a "bump" in a batch size (which may exceed pre-allocation in a 
dedicated separate allocator).




---

[GitHub] drill issue #905: DRILL-1162: Fix OOM for hash join operator when the right ...

[jinfengni]

Github user jinfengni commented on the issue:

https://github.com/apache/drill/pull/905
  
@vvysotskyi , what I'm thinking is not just comparing the ratio of column 
count vs ratio of row count.  

Let's take a step back. This SwapHashJoinVisitor is trying to correct the 
join decision made by either VolcanoPlanner or HepPlanner, in the sense the 
original join order might need put bigger dataset on the build side. The swap 
is needed because we want to avoid high memory requirement on hash join 
operator. The question is how do we define "big dataset". For now, 
SwapHashJoinVisitor simply uses rowCount, which is not sufficient. 

In stead, we probably should add a method `getSelfMemCost` to all `Prel` 
node. For non-blocking operator, it's simply returning either 0 or some 
constant (to hold one single batch). For non-blocking operator such as 
HashJoin, it will return a value proportional to rowCount X columnCount (more 
precisely, total number of bytes per row, considering different column data 
type). 

Same as existing method of `computeSelfCost`, we need 
`getCumulativeMemCost` which will return the cumulative cost for child nodes 
rooted at one `Prel` node.  With this `getSelfMemCost` and 
`getCumulativeMemCost` defined for HashJoin, and a HashJoin with input1, input2 
as inputs, we could estimate cumulative memory cost for HashJoin(input1, 
input2), and HashJoin(input2, input1), and use that as criteria to decide 
whether we have to switch them. 

This idea is not trying to adjust the row count estimation. In stead, it's 
trying to change the criteria where we may think it's necessary to swap, based 
on the observation that we want to do swap only when we want to reduce memory 
requirement for a query. 

Will the above idea work? If it could not address this issue, it's probably 
fine to go with what you proposed. Before we go that option, please give some 
thoughts about the above idea. 




---

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138435187
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -382,19 +390,25 @@ private void delayedSetup() {
 final boolean fallbackEnabled = 
context.getOptions().getOption(ExecConstants.HASHAGG_FALLBACK_ENABLED_KEY).bool_val;
 
 // Set the number of partitions from the configuration (raise to a 
power of two, if needed)
-numPartitions = 
context.getConfig().getInt(ExecConstants.HASHAGG_NUM_PARTITIONS);
-if ( numPartitions == 1 ) {
+numPartitions = 
(int)context.getOptions().getOption(ExecConstants.HASHAGG_NUM_PARTITIONS_VALIDATOR);
+if ( numPartitions == 1 && is2ndPhase  ) { // 1st phase can still do 
early return with 1 partition
   canSpill = false;
   logger.warn("Spilling is disabled due to configuration setting of 
num_partitions to 1");
 }
 numPartitions = BaseAllocator.nextPowerOfTwo(numPartitions); // in 
case not a power of 2
 
-if ( schema == null ) { estMaxBatchSize = 0; } // incoming was an 
empty batch
+if ( schema == null ) { estValuesBatchSize = estOutgoingAllocSize = 
estMaxBatchSize = 0; } // incoming was an empty batch
 else {
   // Estimate the max batch size; should use actual data (e.g. lengths 
of varchars)
   updateEstMaxBatchSize(incoming);
 }
-long memAvail = memoryLimit - allocator.getAllocatedMemory();
+// create "reserved memory" and adjust the memory limit down
+reserveValueBatchMemory = reserveOutgoingMemory = estValuesBatchSize ;
+long newMemoryLimit = allocator.getLimit() - reserveValueBatchMemory - 
reserveOutgoingMemory ;
+long memAvail = newMemoryLimit - allocator.getAllocatedMemory();
+if ( memAvail <= 0 ) { throw new OutOfMemoryException("Too little 
memory available"); }
+allocator.setLimit(newMemoryLimit);
+
--- End diff --

Yes indeed. The only attempt to force a code path is the new testing option 
*use_memory_prediction* which can disable the estimate based prediction (when 
to spill), hence forcing the code path that relies on an OOM (for hash table 
put() ) to take place (one unit test was added for that).
Getting a full code coverage would be ideal, but hard.


---

[jira] [Created] (DRILL-5783) Make code generation in the TopN operator more modular and test it

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5783:
-

 Summary: Make code generation in the TopN operator more modular 
and test it
 Key: DRILL-5783
 URL: https://issues.apache.org/jira/browse/DRILL-5783
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Timothy Farkas
Assignee: Timothy Farkas






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #938: DRILL-5694: Handle HashAgg OOM by spill and retry, ...

[Ben-Zvi]

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/938#discussion_r138433967
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java
 ---
@@ -500,22 +516,45 @@ private void initializeSetup(RecordBatch newIncoming) 
throws SchemaChangeExcepti
*/
   private void updateEstMaxBatchSize(RecordBatch incoming) {
 if ( estMaxBatchSize > 0 ) { return; }  // no handling of a schema (or 
varchar) change
+// Use the sizer to get the input row width and the length of the 
longest varchar column
 RecordBatchSizer sizer = new RecordBatchSizer(incoming);
 logger.trace("Incoming sizer: {}",sizer);
 // An empty batch only has the schema, can not tell actual length of 
varchars
 // else use the actual varchars length, each capped at 50 (to match 
the space allocation)
-estRowWidth = sizer.rowCount() == 0 ? sizer.stdRowWidth() : 
sizer.netRowWidthCap50();
-estMaxBatchSize = estRowWidth * MAX_BATCH_SIZE;
+long estInputRowWidth = sizer.rowCount() == 0 ? sizer.stdRowWidth() : 
sizer.netRowWidthCap50();
 
 // Get approx max (varchar) column width to get better memory 
allocation
-maxColumnWidth = Math.max(sizer.maxSize(), 
VARIABLE_MIN_WIDTH_VALUE_SIZE);
+maxColumnWidth = Math.max(sizer.maxAvgColumnSize(), 
VARIABLE_MIN_WIDTH_VALUE_SIZE);
 maxColumnWidth = Math.min(maxColumnWidth, 
VARIABLE_MAX_WIDTH_VALUE_SIZE);
 
-logger.trace("{} phase. Estimated row width: {}  batch size: {}  
memory limit: {}  max column width: {}",
-
isTwoPhase?(is2ndPhase?"2nd":"1st"):"Single",estRowWidth,estMaxBatchSize,memoryLimit,maxColumnWidth);
+//
+// Calculate the estimated max (internal) batch (i.e. Keys batch + 
Values batch) size
+// (which is used to decide when to spill)
+// Also calculate the values batch size (used as a reserve to overcome 
an OOM)
+//
+Iterator outgoingIter = outContainer.iterator();
+int fieldId = 0;
+while (outgoingIter.hasNext()) {
+  ValueVector vv = outgoingIter.next().getValueVector();
+  MaterializedField mr = vv.getField();
+  int fieldSize = vv instanceof VariableWidthVector ? maxColumnWidth :
+  TypeHelper.getSize(mr.getType());
+  estRowWidth += fieldSize;
+  estOutputRowWidth += fieldSize;
+  if ( fieldId < numGroupByOutFields ) { fieldId++; }
+  else { estValuesRowWidth += fieldSize; }
+}
+// multiply by the max number of rows in a batch to get the final 
estimated max size
+estMaxBatchSize = Math.max(estRowWidth, estInputRowWidth) * 
MAX_BATCH_SIZE;
--- End diff --

Most of these estimates are for internal "worst case".  Only the "outgoing" 
one is for the outgoing batch (which is also for spilling - which is internal).
   Anyway all these estimates have nothing to do with _throttling_ the 
outgoing batch size; that logic was not changed from the original code (likely 
up to MAX_BATCH_SIZE). 
  Making such a change should be a separate project. 



---

[GitHub] drill issue #905: DRILL-1162: Fix OOM for hash join operator when the right ...

[vvysotskyi]

Github user vvysotskyi commented on the issue:

https://github.com/apache/drill/pull/905
  
@jinfengni thanks for looking into this. Completely agree with you that it 
would be better to consider both row and column count. 
Unfortunately, it does not help to fix this issue, since the ratio of 
column count of tables very often much smaller than the ratio of row count 
(actual row count). (`c1/c2 << r2/r1`)
So I guess it makes sense to implement your proposal to consider the column 
count together with the row count in another pull request.


---

[GitHub] drill pull request #930: DRILL-5761: Disable Lilith ClassicMultiplexSocketAp...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/930#discussion_r138408349
  
--- Diff: common/src/test/resources/logback-test.xml ---
@@ -0,0 +1,92 @@
+
+
+
+
+  
+
+  
+true
+1
+true
+${LILITH_HOSTNAME:-localhost}
+  
+
+  
+
+  
+
+
+  %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
+
+  
+
+  

[GitHub] drill pull request #930: DRILL-5761: Disable Lilith ClassicMultiplexSocketAp...

[vrozov]

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/930#discussion_r138407993
  
--- Diff: common/src/test/resources/logback-test.xml ---
@@ -0,0 +1,111 @@
+
+
+
+
+  
+
+  
+true
+1
+true
+${LILITH_HOSTNAME:-localhost}
+  
+
+  
+
+  
+
+
+  %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
+
+  
+
+  
+
+  
+
--- End diff --

Can this logger be defined between line 21 and 22 in a single if condition?


---

[GitHub] drill issue #936: DRILL-5772: Add unit tests to indicate how utf-8 support c...

[arina-ielchiieva]

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/936
  
@paul-rogers, the unit tests just show which influence saffron property has 
on Drill (since people in the community where asking how to enable support 
UTF-8 in Drill), a long with this PR we'll also add description to Drill 
documentation. 

So far Drill relies on Calcite saffron property to determine which charset 
it supports. By default, it's ISO-8859-1. So currently if customer wants to 
query UTF-8 data in Drill, he will get an error (one of the unit tests shows 
it) and needs to override saffron property to proceed. I guess, we don't 
support UTF-8 by default since there are some issues and Drill did not fully 
tested UTF-8 support.


---

[jira] [Created] (DRILL-5782) Web UI: do not attempt build visualized plan when plan is absent

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-5782:
---

 Summary: Web UI: do not attempt build visualized plan when plan is 
absent
 Key: DRILL-5782
 URL: https://issues.apache.org/jira/browse/DRILL-5782
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Arina Ielchiieva
 Attachments: error_when_creating_visualized_plan.JPG

When trying to execute query with incorrect syntax, there is an error on query 
profile page (screenshot attached). Error occurs when we try to build 
visualized plan when plan is absent. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5339) Web UI: flaws in query profile display on error

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5339.
-
Resolution: Fixed

Points 1-4 fixed in the scope of DRILL-5766.

> Web UI: flaws in query profile display on error
> ---
>
> Key: DRILL-5339
> URL: https://issues.apache.org/jira/browse/DRILL-5339
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: error_shift.JPG, fragment_profiles.JPG, 
> verbose_error.JPG, visualized_plan.JPG
>
>
> 1. Error is shifted to the right (screenshot - error_shift.jpg)
> 2. When Fragment Profiles doesn't have any data table, there is offset in 
> overview tab (screenshot - fragment_profiles.jpg)
> 3. Verbose error message tab can be designed like overview tab to be more 
> concise (screenshot - verbose_error.jpg)
> 4. Remove three dots in the end of "Verbose error message..." .
> 5. When visualized plan is empty, there is big offset (screenshot - 
> visualized_plan.jpg)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5338) Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile that they are part of total time duration

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5338.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: add better visual indication for PLANNING, QUEUED, EXECUTION in Query 
> Profile that they are part of total time duration
> ---
>
> Key: DRILL-5338
> URL: https://issues.apache.org/jira/browse/DRILL-5338
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: 1.12.0
>
> Attachments: QueryProfile.JPG
>
>
> Add better visual indication for PLANNING, QUEUED, EXECUTION in Query Profile 
> that they are part of total time duration
> Screenshot is attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5341) Web UI: remove duplicating link to documentation in Options page

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5341.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: remove duplicating link to documentation in Options page
> 
>
> Key: DRILL-5341
> URL: https://issues.apache.org/jira/browse/DRILL-5341
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: doc_link.JPG
>
>
> Remove link to Documentation on Options page (http://localhost:8047/options) 
> as we have the same link on page header. Screenshot - doc_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (DRILL-5346) Web UI: remove link on user in query profile list

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva resolved DRILL-5346.
-
Resolution: Fixed

Fixed in the scope of DRILL-5766

> Web UI: remove link on user in query profile list
> -
>
> Key: DRILL-5346
> URL: https://issues.apache.org/jira/browse/DRILL-5346
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.12.0
>
> Attachments: query_profile_link.JPG
>
>
> Remove link on user column that leads to query profile. Since we don't have 
> user page information, it's better to leave user name without link to avoid 
> confusion. Plus we already have link that leads to query profile on query 
> itself.
> Page - http://localhost:8047/profiles
> Screenshot - query_profile_link.JPG



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] drill pull request #935: DRILL-5766: Fix XSS vulnerabilities in Drill

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/935


---

[jira] [Created] (DRILL-5781) Fix unit test failures to use tests config even if default config is available

[Volodymyr Vysotskyi (JIRA)]

Volodymyr Vysotskyi created DRILL-5781:
--

 Summary: Fix unit test failures to use tests config even if 
default config is available
 Key: DRILL-5781
 URL: https://issues.apache.org/jira/browse/DRILL-5781
 Project: Apache Drill
  Issue Type: Bug
Affects Versions: 1.11.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi


Unit tests fail when they are run with the mapr profile.
Tests failures, connected with the Zookeeper configuration that differs from 
expected:
{noformat}
DrillClientTest>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
Coul...
  TestZookeeperClient.testPutWithMatchingVersion » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testStartingClientEnablesCacheAndEnsuresRootNodeExists » 
IO
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathThrowsDrillRuntimeException » IO Could not 
conf...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathFalseWithVersion » IO Could not configure 
serve...
  TestZookeeperClient.tearDown:86 NullPointer
  TestEphemeralStore.testPutAndGetWorksAntagonistacally » IO Could not 
configure...
  TestEphemeralStore.tearDown:132 NullPointer
  TestZookeeperClient.testGetWithVersion » IO Could not configure server 
because...
  TestZookeeperClient.tearDown:86 NullPointer
  TestEphemeralStore.testStoreRegistersDispatcherAndStartsItsClient » IO Could 
n...
  TestEphemeralStore.tearDown:132 NullPointer
  TestZookeeperClient.testPutWithNonMatchingVersion » IO Could not configure 
ser...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testGetWithEventualConsistencyHitsCache » IO Could not 
con...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutIfAbsentWhenPresent » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathTrueWithVersion » IO Could not configure 
server...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutAndGetWorks » IO Could not configure server 
because...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testPutIfAbsentWhenAbsent » IO Could not configure server 
...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testHasPathWithEventualConsistencyHitsCache » IO Could 
not...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testCreate » IO Could not configure server because SASL 
co...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testDelete » IO Could not configure server because SASL 
co...
  TestZookeeperClient.tearDown:86 NullPointer
  TestZookeeperClient.testEntriesReturnsRelativePaths » IO Could not configure 
s...
  TestZookeeperClient.tearDown:86 NullPointer
TestPStoreProviders>TestWithZookeeper.setUp:32 » Runtime java.io.IOException: 
...
  TestPauseInjection.pauseOnSpecificBit:151 » Runtime java.io.IOException: 
Could...
  TestExceptionInjection.injectionOnSpecificBit:217 » Runtime 
java.io.IOExceptio...

HBaseTestsSuite.initCluster:110 » IO No JAAS configuration section named 
'Serv...
{noformat}

Test failures, connected with Hadoop configuration that differs from expected:
{noformat}
TestInboundImpersonation.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestImpersonationMetadata.setup:58->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestImpersonationDisabledWithMiniDFS.setup:37->BaseTestImpersonation.startMiniDfsCluster:106
 » Runtime
  
TestImpersonationQueries.setup:46->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast

TestHiveStorage>HiveTestBase.generateHive:34 » Runtime 
java.lang.RuntimeExcept...
  TestInfoSchemaOnHiveStorage>HiveTestBase.generateHive:34 » Runtime 
java.lang.R...
  TestInbuiltHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
sett...
  TestSampleHiveUDFs>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
setti...
  
TestStorageBasedHiveAuthorization.setup:109->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  
TestSqlStdBasedAuthorization.setup:72->BaseTestImpersonation.startMiniDfsCluster:80->BaseTestImpersonation.startMiniDfsCluster:111
 » ClassCast
  TestHivePartitionPruning>HiveTestBase.generateHive:35 » ExecutionSetup 
Failure...
  TestViewSupportOnHiveTables.generateHive:35 » ExecutionSetup Failure setting 
u...
  TestHiveProjectPushDown>HiveTestBase.generateHive:35 » ExecutionSetup Failure 
...
{noformat}

Test specific configuration should be added to fix these failures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)