[jira] [Updated] (DRILL-6320) Don't Allow Javadoc comments for license headers

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6320:

Description: 
Currently some license headers are in a javadoc comment instead of a normal 
comment. This is not good since we don't want to include spurious license 
headers in java docs when we publish them. The fix would be to change the rat 
plugin configuration to not allow java doc comments.

 

*For documentation*

Min required maven version has changed to {{Apache Maven 3.3.1}}.
Update - https://drill.apache.org/docs/compiling-drill-from-source/

  was:
Currently some license headers are in a javadoc comment instead of a normal 
comment. This is not good since we don't want to include spurious license 
headers in java docs when we publish them. The fix would be to change the rat 
plugin configuration to not allow java doc comments.

 

*For documentation*

Min required maven version has changed.


> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed to {{Apache Maven 3.3.1}}.
> Update - https://drill.apache.org/docs/compiling-drill-from-source/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6320) Don't Allow Javadoc comments for license headers

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6320:

Labels: doc-impacting ready-to-commit  (was: ready-to-commit)

> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6289) Cluster view should show more relevant information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439786#comment-16439786
 ] 

ASF GitHub Bot commented on DRILL-6289:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r181830465
  
--- Diff: 
common/src/main/java/org/apache/drill/exec/metrics/CpuGaugeSet.java ---
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.metrics;
+
+import java.lang.management.OperatingSystemMXBean;
+import java.util.HashMap;
+import java.util.Map;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Metric;
+import com.codahale.metrics.MetricSet;
+
+/**
+ * Creates a Cpu GaugeSet
+ */
+class CpuGaugeSet implements MetricSet {
--- End diff --

I am not sure how useful `getProcessCpuLoad` alone will be, but in addition 
with total system load it can give the idea about how Drill and other services 
on node are using the CPU.  If you plan to include another metric then in that 
case `CpuGaugeSet` class is fine. Thanks for clarification.


> Cluster view should show more relevant information
> --
>
> Key: DRILL-6289
> URL: https://issues.apache.org/jira/browse/DRILL-6289
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When fixing DRILL-6224, I noticed that the same information can be very 
> useful to have in the cluster view shown on a Drillbit's homepage. 
> The proposal is to show the following:
> # Heap Memory in use
> # Direct Memory (actively) in use - Since we're not able to get the total 
> memory held by Netty at the moment, but only what is currently allocated to 
> running queries
> # Process CPU
> # Average (System) Load Factor 
> Information such as the port numbers don't help much during general cluster 
> health, so it might be worth removing this information if more real-estate is 
> needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439457#comment-16439457
 ] 

ASF GitHub Bot commented on DRILL-6270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1210#discussion_r181732701
  
--- Diff: distribution/src/resources/runbit ---
@@ -65,6 +65,47 @@ drill_rotate_log ()
 fi
 }
 
+args=( $@ )
+RBARGS=()
--- End diff --

Does `RBARGS` have some meaning? Maybe it better to give clearer naming?
Also please add comment describing that you remove debug string from 
original args but leave all other args.


> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6289) Cluster view should show more relevant information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439797#comment-16439797
 ] 

ASF GitHub Bot commented on DRILL-6289:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1203#discussion_r181834064
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -283,20 +308,114 @@
alert(errorThrown);
},
success: function(data) {
-  alert(data["response"]);
+   alert(data["response"]);

button.prop('disabled',true).css('opacity',0.5);
}
   });
   }
   }
-  
+
+  function remoteShutdown(button,host) {
+  var url = location.protocol + "//" + host + 
"/gracefulShutdown";
+  var result = $.ajax({
+type: 'POST',
+url: url,
+contentType : 'text/plain',
+complete: function(data) {
+alert(data.responseJSON["response"]);
+button.prop('disabled',true).css('opacity',0.5);
+}
+  });
+  }
+  
+
+  function popOutRemoteDbitUI(dbitHost, dbitPort) {
--- End diff --

We can't put this and the remaining functions within the IF block since the 
localhost operations are still valid. i.e. an authenticated client on a 
specific Drillbit can ping that Drillbit for refresh of the metrics. Only 
shutdown function needs to be managed with the IF block


> Cluster view should show more relevant information
> --
>
> Key: DRILL-6289
> URL: https://issues.apache.org/jira/browse/DRILL-6289
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When fixing DRILL-6224, I noticed that the same information can be very 
> useful to have in the cluster view shown on a Drillbit's homepage. 
> The proposal is to show the following:
> # Heap Memory in use
> # Direct Memory (actively) in use - Since we're not able to get the total 
> memory held by Netty at the moment, but only what is currently allocated to 
> running queries
> # Process CPU
> # Average (System) Load Factor 
> Information such as the port numbers don't help much during general cluster 
> health, so it might be worth removing this information if more real-estate is 
> needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[Anton Gozhiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Gozhiy updated DRILL-6270:

Description: 
Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
with standard java remote debug options with the ability to override port.

{noformat}
Usage: --debug:[parameter1=value,parameter2=value]
Optional parameters:
port=[port_number] - debug port number
suspend=[y/n] - pause until the IDE connects
{noformat}

Examples: 
{noformat}
drillbit.sh start --debug:port=50001,suspend=y
drill-embedded --debug:port=50001,suspend=y
sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
{noformat}

Also should work with sqlline.bat, but in this case the debug parameters should 
be wrapped up with quotes, like:
{noformat}
sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
{noformat}

  was:
Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
with standard java remote debug options with the ability to override port.

Example: drillbit.sh start - 50001


> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Minor
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6331) Parquet filter pushdown does not support the native hive reader

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-6331:
---

 Summary: Parquet filter pushdown does not support the native hive 
reader
 Key: DRILL-6331
 URL: https://issues.apache.org/jira/browse/DRILL-6331
 Project: Apache Drill
  Issue Type: Improvement
  Components: Storage - Hive
Affects Versions: 1.13.0
Reporter: Arina Ielchiieva
Assignee: Arina Ielchiieva
 Fix For: 1.14.0


Initially HiveDrillNativeParquetGroupScan was based mainly on HiveScan, the 
core difference between them was
that HiveDrillNativeParquetScanBatchCreator was creating ParquetRecordReader 
instead of HiveReader.
This allowed to read Hive parquet files using Drill native parquet reader but 
did not expose Hive data to Drill optimizations.
For example, filter push down, limit push down, count to direct scan 
optimizations.
Hive code had to be refactored to use the same interfaces as ParquestGroupScan 
in order to be exposed to such optimizations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439657#comment-16439657
 ] 

Paul Rogers commented on DRILL-6270:


Please see the note above. This should not be debug specific. We should not be 
in the business of generating the Java debug flags. As noted, better to provide 
a general method to pass options to the JVM command line. Then the user can 
decide the correct debug settings for their case. At the very least, they can 
Google how to set the flags and copy/paste that into the Drill command line.

> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439667#comment-16439667
 ] 

ASF GitHub Bot commented on DRILL-6270:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1210#discussion_r181799954
  
--- Diff: distribution/src/resources/runbit ---
@@ -65,6 +65,47 @@ drill_rotate_log ()
 fi
 }
 
+args=( $@ )
+RBARGS=()
+for (( i=0; i < ${#args[@]}; i++ )); do
+  case "${args[i]}" in
+  --debug*)
+  DEBUG=true
+  DEBUG_STRING=`expr "${args[i]}" : '.*:\(.*\)'`
+  ;;
+   *) RBARGS+=("${args[i]}");;
+  esac
+done
+
+# Enables remote debug if requested
+# Usage: --debug:[parameter1=value,parameter2=value]
+# Optional parameters:
+# port=[port_number] - debug port number
+# suspend=[y/n] - pause until the IDE connects
+
+if [ $DEBUG ]; then
+  debug_params=( $(echo $DEBUG_STRING | grep -o '[^,"]*') )
+  for param in ${debug_params[@]}; do
+case $param in
+port*)
+  DEBUG_PORT=`expr "$param" : '.*=\(.*\)'`
+  ;;
+suspend*)
+  DEBUG_SUSPEND=`expr "$param" : '.*=\(.*\)'`
+  ;;
+esac
+  done
+
+  if [ -z $DEBUG_PORT ]; then
+DEBUG_PORT=5
+  fi
+  if [ -z $DEBUG_SUSPEND ]; then
+DEBUG_SUSPEND='n'
+  fi
+
+  JAVA_DEBUG="-Xdebug -Xnoagent 
-Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=$DEBUG_SUSPEND"
+fi
--- End diff --

This is overly complex and puts Drill in the business of tracking the 
arguments that the JVM wants to enable debugging. Those flags changed in the 
past and may change again.

Would recommend instead:

```
--jvm -anyArgumentYouWant
```

Multiple `--jvm` arguments can appear. Simply append this to the the 
`DRILL_JAVA_OPTS` variable. Very simple.


> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6270:

Reviewer: Arina Ielchiieva

> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6332) DrillbitStartupException: Failure while initializing values in Drillbit

[Hari Sekhon (JIRA)]

Hari Sekhon created DRILL-6332:
--

 Summary: DrillbitStartupException: Failure while initializing 
values in Drillbit
 Key: DRILL-6332
 URL: https://issues.apache.org/jira/browse/DRILL-6332
 Project: Apache Drill
  Issue Type: Improvement
  Components:  Server
Affects Versions: 1.10.0
Reporter: Hari Sekhon


Improvement request to make this error more specific so we can tell what is 
causing it:
{code:java}

==> /opt/mapr/drill/drill-1.10.0/logs/drillbit.out <==
Exception in thread "main" 
org.apache.drill.exec.exception.DrillbitStartupException: Failure while 
initializing values in Drillbit.
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
Caused by: java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
at 
org.apache.drill.common.KerberosUtil.splitPrincipalIntoParts(KerberosUtil.java:59)
at 
org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:130)
at 
org.apache.drill.exec.server.BootStrapContext.(BootStrapContext.java:77)
at org.apache.drill.exec.server.Drillbit.(Drillbit.java:94)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
... 2 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439290#comment-16439290
 ] 

ASF GitHub Bot commented on DRILL-6270:
---

GitHub user agozhiy opened a pull request:

https://github.com/apache/drill/pull/1210

DRILL-6270: Add debug startup option flag for drill in embedded and s…

…erver mode

Works with the drillbit.sh, sqlline and sqlline.bat:
 Usage: --debug:[parameter1=value,parameter2=value]
 Optional parameters:
 port=[port_number] - debug port number
 suspend=[y/n] - pause until the IDE connects

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agozhiy/drill DRILL-6270

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1210.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1210


commit 5206268568a43864f6b0d144b56d3033c17d8031
Author: Anton Gozhiy 
Date:   2018-03-27T10:29:14Z

DRILL-6270: Add debug startup option flag for drill in embedded and server 
mode

Works with the drillbit.sh, sqlline and sqlline.bat:
 Usage: --debug:[parameter1=value,parameter2=value]
 Optional parameters:
 port=[port_number] - debug port number
 suspend=[y/n] - pause until the IDE connects




> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Minor
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6270:

Priority: Major  (was: Minor)

> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Major
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6270) Add debug startup option flag for drill in embedded and server mode

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6270:

Fix Version/s: 1.14.0

> Add debug startup option flag for drill in embedded and server mode
> ---
>
> Key: DRILL-6270
> URL: https://issues.apache.org/jira/browse/DRILL-6270
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Volodymyr Tkach
>Assignee: Anton Gozhiy
>Priority: Minor
> Fix For: 1.14.0
>
>
> Add possibility to run sqlline.sh and drillbit.sh scripts with -- 
> with standard java remote debug options with the ability to override port.
> {noformat}
> Usage: --debug:[parameter1=value,parameter2=value]
> Optional parameters:
> port=[port_number] - debug port number
> suspend=[y/n] - pause until the IDE connects
> {noformat}
> Examples: 
> {noformat}
> drillbit.sh start --debug:port=50001,suspend=y
> drill-embedded --debug:port=50001,suspend=y
> sqlline -u "jdbc:drill:zk=[zk_address];" --debug:port=50001,suspend=y
> {noformat}
> Also should work with sqlline.bat, but in this case the debug parameters 
> should be wrapped up with quotes, like:
> {noformat}
> sqlline.bat -u "jdbc:drill:zk=local;" --debug:"port=50001,suspend=y"
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439921#comment-16439921
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181858695
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -118,105 +127,114 @@ public PartitionOutgoingBatch 
getOutgoingBatches(int index) {
 return null;
   }
 
-  @VisibleForTesting
-  protected List getPartitioners() {
+  List getPartitioners() {
 return partitioners;
   }
 
   /**
* Helper to execute the different methods wrapped into same logic
* @param iface
-   * @throws IOException
+   * @throws ExecutionException
*/
-  protected void executeMethodLogic(final GeneralExecuteIface iface) 
throws IOException {
-if (partitioners.size() == 1 ) {
-  // no need for threads
-  final OperatorStats localStatsSingle = 
partitioners.get(0).getStats();
-  localStatsSingle.clear();
-  localStatsSingle.startProcessing();
+  @VisibleForTesting
+  void executeMethodLogic(final GeneralExecuteIface iface) throws 
ExecutionException {
+// To simulate interruption of main fragment thread and interrupting 
the partition threads, create a
+// CountDownInject latch. Partitioner threads await on the latch and 
main fragment thread counts down or
+// interrupts waiting threads. This makes sure that we are actually 
interrupting the blocked partitioner threads.
+try (CountDownLatchInjection testCountDownLatch = 
injector.getLatch(context.getExecutionControls(), "partitioner-sender-latch")) {
--- End diff --

I'm not sure that we should be using the injector to create a count down 
latch here. My understanding is that we have to define a 
`partitioner-sender-latch` injection site on the 
`"drill.exec.testing.controls"` property and it is intended only for testing. 
See ControlsInjectionUtil.createLatch(). The default value for 
`drill.exec.testing.controls` is empty so the getLatch method would return a 
Noop latch since `partitioner-sender-latch` is undefined. Since we always want 
to create a count down latch here (not just for testing) shouldn't we directly 
create one?


> PartitionerDecorator may close partitioners while CustomRunnable are active 
> during query cancellation
> -
>
> Key: DRILL-6295
> URL: https://issues.apache.org/jira/browse/DRILL-6295
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> During query cancellation, in case 
> {{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the 
> {{latch}}), the wait will be interrupted and {{Futures}} cancelled, but there 
> is no guarantee that all {{CustomRunnable}} terminate before returning from 
> {{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and 
> outgoing batches are cleared, leading to clearing of underlying {{Vectors}} 
> and {{DrillBufs}}. This eventually causes unallocated memory access and JVM 
> crash as {{CustomRunnable}} may execute after income/outgoing batches are 
> cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6320) Don't Allow Javadoc comments for license headers

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas updated DRILL-6320:
--
Labels: ready-to-commit  (was: )

> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6301) Parquet Performance Analysis

[salim achouche (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439954#comment-16439954
 ] 

salim achouche commented on DRILL-6301:
---

[~parthc] / [~vrozov], 

 

Can you please review the benchmark tests 
[drill-jmh|https://github.com/sachouche/drill-jmh] and the associated test 
results listed in this 
[document|https://docs.google.com/document/d/1BSNem_ItP-Vxlr6auSP_iwwOLM9rwWZYxGwCsXi-IE8/edit?usp=sharing]?

Thanks!

 

> Parquet Performance Analysis
> 
>
> Key: DRILL-6301
> URL: https://issues.apache.org/jira/browse/DRILL-6301
> Project: Apache Drill
>  Issue Type: Task
>  Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> _*Description -*_
>  * DRILL-5846 is meant to improve the Flat Parquet reader performance
>  * The associated implementation resulted in a 2x - 4x performance improvement
>  * Though during the review process ([pull 
> request|[https://github.com/apache/drill/pull/1060])] few key questions arised
>  
> *_Intermediary Processing via Direct Memory vs Byte Arrays_*
>  * The main reasons for using byte arrays for intermediary processing is to 
> a) avoid the high cost of the DrillBuf checks (especially the reference 
> counting) and b) benefit from some observed Java optimizations when accessing 
> byte arrays
>  * Starting with version 1.12.0, the DrillBuf enablement checks have been 
> refined so that memory access and reference counting checks can be enabled 
> independently
>  * Benchmarking of Java's Direct Memory unsafe method using JMH indicates the 
> performance gap between heap vs direct memory  is very narrow except for few 
> use-cases
>  * There are also concerns that the extra copy step (from direct memory into 
> byte arrays) will have a negative effect on performance; note that this 
> overhead was not observed using Intel's Vtune as the intermediary buffer were 
> a) pinned to a single CPU, b) reused, and c) small enough to remain in the L1 
> cache during columnar processing.
> _*Goal*_ 
>  * The Flat Parquet reader is amongst the few Drill columnar operators
>  * It is imperative that we agree on the most optimal processing pattern so 
> that the decisions that we take within this Jira are not only applied to 
> Parquet but to all Drill columnar operators   
> _*Methodology*_ 
>  # Assess the performance impact of using intermediary byte arrays (as 
> described above)
>  # Prototype a solution using Direct Memory and DrillBuf checks off, access 
> checks on, all checks on
>  # Make an educated decision on which processing pattern should be adopted
>  # Decide whether it is ok to use Java's unsafe API (and through what 
> mechanism) on byte arrays (when the use of byte arrays is a necessity)
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439975#comment-16439975
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181871311
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -118,105 +127,114 @@ public PartitionOutgoingBatch 
getOutgoingBatches(int index) {
 return null;
   }
 
-  @VisibleForTesting
-  protected List getPartitioners() {
+  List getPartitioners() {
 return partitioners;
   }
 
   /**
* Helper to execute the different methods wrapped into same logic
* @param iface
-   * @throws IOException
+   * @throws ExecutionException
*/
-  protected void executeMethodLogic(final GeneralExecuteIface iface) 
throws IOException {
-if (partitioners.size() == 1 ) {
-  // no need for threads
-  final OperatorStats localStatsSingle = 
partitioners.get(0).getStats();
-  localStatsSingle.clear();
-  localStatsSingle.startProcessing();
+  @VisibleForTesting
+  void executeMethodLogic(final GeneralExecuteIface iface) throws 
ExecutionException {
+// To simulate interruption of main fragment thread and interrupting 
the partition threads, create a
+// CountDownInject latch. Partitioner threads await on the latch and 
main fragment thread counts down or
+// interrupts waiting threads. This makes sure that we are actually 
interrupting the blocked partitioner threads.
+try (CountDownLatchInjection testCountDownLatch = 
injector.getLatch(context.getExecutionControls(), "partitioner-sender-latch")) {
--- End diff --

I see thx.


> PartitionerDecorator may close partitioners while CustomRunnable are active 
> during query cancellation
> -
>
> Key: DRILL-6295
> URL: https://issues.apache.org/jira/browse/DRILL-6295
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> During query cancellation, in case 
> {{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the 
> {{latch}}), the wait will be interrupted and {{Futures}} cancelled, but there 
> is no guarantee that all {{CustomRunnable}} terminate before returning from 
> {{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and 
> outgoing batches are cleared, leading to clearing of underlying {{Vectors}} 
> and {{DrillBufs}}. This eventually causes unallocated memory access and JVM 
> crash as {{CustomRunnable}} may execute after income/outgoing batches are 
> cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5900) Regression: TPCH query encounters random IllegalStateException: Memory was leaked by query

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439930#comment-16439930
 ] 

Timothy Farkas commented on DRILL-5900:
---

Rebased onto the latest master [~rhou] please try to reproduce when you have 
time.

> Regression: TPCH query encounters random IllegalStateException: Memory was 
> leaked by query
> --
>
> Key: DRILL-5900
> URL: https://issues.apache.org/jira/browse/DRILL-5900
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.11.0
>Reporter: Robert Hou
>Assignee: Timothy Farkas
>Priority: Blocker
> Attachments: 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f.sys.drill, 
> drillbit.log.node81, drillbit.log.node88
>
>
> This is a random failure in the TPCH-SF100-baseline run.  The test is 
> /root/drillAutomation/framework-master/framework/resources/Advanced/tpch/tpch_sf1/original/parquet/query17.sql.
>   This test has passed before.
> TPCH query 6:
> {noformat}
> SELECT
>   SUM(L.L_EXTENDEDPRICE) / 7.0 AS AVG_YEARLY
> FROM
>   lineitem L,
>   part P
> WHERE
>   P.P_PARTKEY = L.L_PARTKEY
>   AND P.P_BRAND = 'BRAND#13'
>   AND P.P_CONTAINER = 'JUMBO CAN'
>   AND L.L_QUANTITY < (
> SELECT
>   0.2 * AVG(L2.L_QUANTITY)
> FROM
>   lineitem L2
> WHERE
>   L2.L_PARTKEY = P.P_PARTKEY
>   )
> {noformat}
> Error is:
> {noformat}
> 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:8:2] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (2097152)
> Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 
> (res/actual/peak/limit)
> Fragment 8:2
> [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (2097152)
> Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 
> (res/actual/peak/limit)
> Fragment 8:2
> [Error Id: f21a2560-7259-4e13-88c2-9bac29e2930a on atsqa6c88.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_51]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (2097152)
> Allocator(op:8:2:6:ParquetRowGroupScan) 100/0/7675904/100 
> (res/actual/peak/limit)
> at 
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:519) 
> ~[drill-memory-base-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:435)
>  ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:424) 
> ~[drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:324)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
>  [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
> ... 5 common frames omitted
> 2017-10-23 10:34:55,989 [2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:frag:6:0] INFO  
> o.a.d.e.w.f.FragmentStatusReporter - 
> 2611d7c0-b0c9-a93e-c64d-a4ef8f4baf8f:6:0: State to report: FINISHED

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439962#comment-16439962
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181865979
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -118,105 +127,114 @@ public PartitionOutgoingBatch 
getOutgoingBatches(int index) {
 return null;
   }
 
-  @VisibleForTesting
-  protected List getPartitioners() {
+  List getPartitioners() {
 return partitioners;
   }
 
   /**
* Helper to execute the different methods wrapped into same logic
* @param iface
-   * @throws IOException
+   * @throws ExecutionException
*/
-  protected void executeMethodLogic(final GeneralExecuteIface iface) 
throws IOException {
-if (partitioners.size() == 1 ) {
-  // no need for threads
-  final OperatorStats localStatsSingle = 
partitioners.get(0).getStats();
-  localStatsSingle.clear();
-  localStatsSingle.startProcessing();
+  @VisibleForTesting
+  void executeMethodLogic(final GeneralExecuteIface iface) throws 
ExecutionException {
+// To simulate interruption of main fragment thread and interrupting 
the partition threads, create a
+// CountDownInject latch. Partitioner threads await on the latch and 
main fragment thread counts down or
+// interrupts waiting threads. This makes sure that we are actually 
interrupting the blocked partitioner threads.
+try (CountDownLatchInjection testCountDownLatch = 
injector.getLatch(context.getExecutionControls(), "partitioner-sender-latch")) {
--- End diff --

The `testCountDownLatch` is used only for testing and initialized to 1. The 
wait is on `count`. 


> PartitionerDecorator may close partitioners while CustomRunnable are active 
> during query cancellation
> -
>
> Key: DRILL-6295
> URL: https://issues.apache.org/jira/browse/DRILL-6295
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> During query cancellation, in case 
> {{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the 
> {{latch}}), the wait will be interrupted and {{Futures}} cancelled, but there 
> is no guarantee that all {{CustomRunnable}} terminate before returning from 
> {{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and 
> outgoing batches are cleared, leading to clearing of underlying {{Vectors}} 
> and {{DrillBufs}}. This eventually causes unallocated memory access and JVM 
> crash as {{CustomRunnable}} may execute after income/outgoing batches are 
> cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6320) Don't Allow Javadoc comments for license headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439871#comment-16439871
 ] 

ASF GitHub Bot commented on DRILL-6320:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1207
  
+1, thanks for making the changes!


> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6320) Don't Allow Javadoc comments for license headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439865#comment-16439865
 ] 

ASF GitHub Bot commented on DRILL-6320:
---

Github user ilooner commented on the issue:

https://github.com/apache/drill/pull/1207
  
@arina-ielchiieva Please let me know if you have any comments or if things 
look good.


> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439872#comment-16439872
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181851002
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -262,68 +280,124 @@ public FlushBatchesHandlingClass(boolean 
isLastBatch, boolean schemaChanged) {
 }
 
 @Override
-public void execute(Partitioner part) throws IOException {
+public void execute(Partitioner part) throws IOException, 
InterruptedException {
   part.flushOutgoingBatches(isLastBatch, schemaChanged);
 }
   }
 
   /**
-   * Helper class to wrap Runnable with customized naming
-   * Exception handling
+   * Helper class to wrap Runnable with cancellation and waiting for 
completion support
*
*/
-  private static class CustomRunnable implements Runnable {
+  private static class PartitionerTask implements Runnable {
+
+private enum STATE {
+  NEW,
+  COMPLETING,
+  NORMAL,
+  EXCEPTIONAL,
+  CANCELLED,
+  INTERRUPTING,
+  INTERRUPTED
+}
+
+private final AtomicReference state;
+private final AtomicReference runner;
+private final PartitionerDecorator partitionerDecorator;
+private final AtomicInteger count;
 
-private final String parentThreadName;
-private final CountDownLatch latch;
 private final GeneralExecuteIface iface;
-private final Partitioner part;
+private final Partitioner partitioner;
 private CountDownLatchInjection testCountDownLatch;
 
-private volatile IOException exp;
+private volatile ExecutionException exception;
 
-public CustomRunnable(final String parentThreadName, final 
CountDownLatch latch, final GeneralExecuteIface iface,
-final Partitioner part, CountDownLatchInjection 
testCountDownLatch) {
-  this.parentThreadName = parentThreadName;
-  this.latch = latch;
+public PartitionerTask(PartitionerDecorator partitionerDecorator, 
GeneralExecuteIface iface, Partitioner partitioner, AtomicInteger count, 
CountDownLatchInjection testCountDownLatch) {
+  state = new AtomicReference<>(STATE.NEW);
+  runner = new AtomicReference<>();
+  this.partitionerDecorator = partitionerDecorator;
   this.iface = iface;
-  this.part = part;
+  this.partitioner = partitioner;
+  this.count = count;
   this.testCountDownLatch = testCountDownLatch;
 }
 
 @Override
 public void run() {
-  // Test only - Pause until interrupted by fragment thread
+  final Thread thread = Thread.currentThread();
+  Preconditions.checkState(runner.compareAndSet(null, thread),
+  "PartitionerTask can be executed only once.");
+  if (state.get() == STATE.NEW) {
+final String name = thread.getName();
+thread.setName(String.format("Partitioner-%s-%d", 
partitionerDecorator.thread.getName(), thread.getId()));
+final OperatorStats localStats = partitioner.getStats();
+localStats.clear();
+localStats.startProcessing();
+ExecutionException executionException = null;
+try {
+  // Test only - Pause until interrupted by fragment thread
+  testCountDownLatch.await();
+  iface.execute(partitioner);
+} catch (InterruptedException e) {
+  if (state.compareAndSet(STATE.NEW, STATE.INTERRUPTED)) {
+logger.warn("Partitioner Task interrupted during the run", e);
+  }
+} catch (Throwable t) {
+  executionException = new ExecutionException(t);
+}
+if (state.compareAndSet(STATE.NEW, STATE.COMPLETING)) {
+  if (executionException == null) {
+localStats.stopProcessing();
+state.lazySet(STATE.NORMAL);
+  } else {
+exception = executionException;
+state.lazySet(STATE.EXCEPTIONAL);
+  }
+}
+if (count.decrementAndGet() == 0) {
+  LockSupport.unpark(partitionerDecorator.thread);
+}
+thread.setName(name);
+  }
+  runner.set(null);
+  while (state.get() == STATE.INTERRUPTING) {
+Thread.yield();
+  }
+  // Clear interrupt flag
   try {
-testCountDownLatch.await();
-  } catch (final InterruptedException e) {
-logger.debug("Test 

[jira] [Commented] (DRILL-6289) Cluster view should show more relevant information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439987#comment-16439987
 ] 

ASF GitHub Bot commented on DRILL-6289:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1203
  
Updated with an additional set of changes. 
* Added CPU metrics (obtained from  
[OperatingSystemMXBean.getProcessCpuLoad()](https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad())
 )
* Added uptime information so that you know if a Drillbit ever has a 
different start time.
* Grey-out a button (disabled, i.e.) if the node is offline or remote & 
requires HTTPS
* Additional changes based on comments
Screenshot

![image](https://user-images.githubusercontent.com/4335237/38833357-1f14b4d6-417a-11e8-98e4-1462628e64bc.png)




> Cluster view should show more relevant information
> --
>
> Key: DRILL-6289
> URL: https://issues.apache.org/jira/browse/DRILL-6289
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When fixing DRILL-6224, I noticed that the same information can be very 
> useful to have in the cluster view shown on a Drillbit's homepage. 
> The proposal is to show the following:
> # Heap Memory in use
> # Direct Memory (actively) in use - Since we're not able to get the total 
> memory held by Netty at the moment, but only what is currently allocated to 
> running queries
> # Process CPU
> # Average (System) Load Factor 
> Information such as the port numbers don't help much during general cluster 
> health, so it might be worth removing this information if more real-estate is 
> needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6118) Handle item star columns during project / filter push down and directory pruning

[Bridget Bevens (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440040#comment-16440040
 ] 

Bridget Bevens commented on DRILL-6118:
---

Documentation is updated for this JIRA: 
[https://drill.apache.org/docs/parquet-filter-pushdown/] 

> Handle item star columns during project  /  filter push down and directory 
> pruning
> --
>
> Key: DRILL-6118
> URL: https://issues.apache.org/jira/browse/DRILL-6118
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.13.0
>
>
> Project push down, filter push down and partition pruning does not work with 
> dynamically expanded column with is represented as star in ITEM operator: 
> _ITEM($0, 'column_name')_ where $0 is a star.
>  This often occurs when view, sub-select or cte with star is issued.
>  To solve this issue we can create {{DrillFilterItemStarReWriterRule}} which 
> will rewrite such ITEM operator before filter push down and directory 
> pruning. For project into scan push down logic will be handled separately in 
> already existing rule {{DrillPushProjectIntoScanRule}}. Basically, we can 
> consider the following queries the same: 
>  {{select col1 from t}}
>  {{select col1 from (select * from t)}}
> *Use cases*
> Since item star columns where not considered during project / filter push 
> down and directory pruning, push down and pruning did not happen. This was 
> causing Drill to read all columns from file (when only several are needed) or 
> ready all files instead. Views with star query is the most common example. 
> Such behavior significantly degrades performance for item star queries 
> comparing to queries without item star.
> *EXAMPLES*
> *Data set* 
> will create table with three files each in dedicated sub-folder:
> {noformat}
> use dfs.tmp;
> create table `order_ctas/t1` as select cast(o_orderdate as date) as 
> o_orderdate from cp.`tpch/orders.parquet` where o_orderdate between date 
> '1992-01-01' and date '1992-01-03';
> create table `order_ctas/t2` as select cast(o_orderdate as date) as 
> o_orderdate from cp.`tpch/orders.parquet` where o_orderdate between date 
> '1992-01-04' and date '1992-01-06';
> create table `order_ctas/t3` as select cast(o_orderdate as date) as 
> o_orderdate from cp.`tpch/orders.parquet` where o_orderdate between date 
> '1992-01-07' and date '1992-01-09';
> {noformat}
> *Filter push down*
> {{select * from order_ctas where o_orderdate = date '1992-01-01'}} will read 
> only one file
> {noformat}
> 00-00Screen
> 00-01  Project(**=[$0])
> 00-02Project(T1¦¦**=[$0])
> 00-03  SelectionVectorRemover
> 00-04Filter(condition=[=($1, 1992-01-01)])
> 00-05  Project(T1¦¦**=[$0], o_orderdate=[$1])
> 00-06Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath [path=/tmp/order_ctas/t1/0_0_0.parquet]], 
> selectionRoot=/tmp/order_ctas, numFiles=1, numRowGroups=1, 
> usedMetadataFile=false, columns=[`**`]]])
> {noformat}
> {{select * from (select * from order_ctas) where o_orderdate = date 
> '1992-01-01'}} will ready all three files
> {noformat}
> 00-00Screen
> 00-01  Project(**=[$0])
> 00-02SelectionVectorRemover
> 00-03  Filter(condition=[=(ITEM($0, 'o_orderdate'), 1992-01-01)])
> 00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=/tmp/order_ctas/t1/0_0_0.parquet], ReadEntryWithPath 
> [path=/tmp/order_ctas/t2/0_0_0.parquet], ReadEntryWithPath 
> [path=/tmp/order_ctas/t3/0_0_0.parquet]], selectionRoot=/tmp/order_ctas, 
> numFiles=3, numRowGroups=3, usedMetadataFile=false, columns=[`**`]]])
> {noformat}
> *Directory pruning*
> {{select * from order_ctas where dir0 = 't1'}} will read data only from one 
> folder
> {noformat}
> 00-00Screen
> 00-01  Project(**=[$0])
> 00-02Project(**=[$0])
> 00-03  Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=/tmp/order_ctas/t1/0_0_0.parquet]], selectionRoot=/tmporder_ctas, 
> numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`**`]]])
> {noformat}
> {{select * from (select * from order_ctas) where dir0 = 't1'}} will read 
> content of all three folders
> {noformat}
> 00-00Screen
> 00-01  Project(**=[$0])
> 00-02SelectionVectorRemover
> 00-03  Filter(condition=[=(ITEM($0, 'dir0'), 't1')])
> 00-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath 
> [path=/tmp/order_ctas/t1/0_0_0.parquet], ReadEntryWithPath 
> [path=/tmp/order_ctas/t2/0_0_0.parquet], ReadEntryWithPath 
> [path=/tmp/order_ctas/t3/0_0_0.parquet]], 

[jira] [Closed] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas closed DRILL-6003.
-
Resolution: Cannot Reproduce

> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled 
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build  Test
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Abhishek Girish
>Assignee: Timothy Farkas
>Priority: Major
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)): 
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException: 
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: 
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
>   at 
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567) 
> ~[classes/:na]
>   at 
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338) 
> ~[test-classes/:na]
>   at 
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147) 
> ~[test-classes/:na]
>   at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139) 
> ~[test-classes/:na]
>   at 
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
>  ~[test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: 
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
>   at 
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468) 
> ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102) 
> ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
>  ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
>  ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> 

[jira] [Commented] (DRILL-6003) Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled fails with FUNCTION ERROR: Failure reading Function class.

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440116#comment-16440116
 ] 

Timothy Farkas commented on DRILL-6003:
---

This error has not appeared for the past few months. I am marking it not 
reproducable.

> Unit test TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled 
> fails with FUNCTION ERROR: Failure reading Function class.
> --
>
> Key: DRILL-6003
> URL: https://issues.apache.org/jira/browse/DRILL-6003
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build  Test
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Abhishek Girish
>Assignee: Timothy Farkas
>Priority: Major
>
> {code}
> 14:05:23.170 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 
> B(1 B), h: 229.7 MiB(1.1 GiB), nh: 187.0 KiB(73.2 MiB)): 
> testLazyInitWhenDynamicUdfSupportIsDisabled(org.apache.drill.TestDynamicUDFSupport)
> org.apache.drill.exec.rpc.RpcException: 
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: 
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
>   at 
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:865)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:567) 
> ~[classes/:na]
>   at 
> org.apache.drill.test.BaseTestQuery.testRunAndReturn(BaseTestQuery.java:338) 
> ~[test-classes/:na]
>   at 
> org.apache.drill.test.BaseTestQuery$ClassicTestServices.testRunAndReturn(BaseTestQuery.java:276)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:830)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:484)
>  ~[test-classes/:na]
>   at 
> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:147) 
> ~[test-classes/:na]
>   at org.apache.drill.test.TestBuilder.go(TestBuilder.java:139) 
> ~[test-classes/:na]
>   at 
> org.apache.drill.TestDynamicUDFSupport.testLazyInitWhenDynamicUdfSupportIsDisabled(TestDynamicUDFSupport.java:506)
>  ~[test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
> org.apache.drill.common.exceptions.UserRemoteException: FUNCTION ERROR: 
> Failure reading Function class.
> Function Class com.drill.udf.CustomLowerFunction
> Fragment 0:0
> [Error Id: 1d6ea0e5-fd65-4622-924d-d196defaedc8 on 10.10.104.57:31010]
>   at 
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:468) 
> ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:102) 
> ~[classes/:na]
>   at 
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244) 
> ~[drill-rpc-1.12.0.jar:1.12.0]
>   at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
>  ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
>  ~[netty-handler-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>  ~[netty-transport-4.0.48.Final.jar:4.0.48.Final]
>   at 
> 

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440065#comment-16440065
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181894189
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
 ---
@@ -161,8 +161,11 @@ public OperatorStats getStats() {
* @param schemaChanged  true if the schema has changed
*/
   @Override
-  public void flushOutgoingBatches(boolean isLastBatch, boolean 
schemaChanged) throws IOException {
+  public void flushOutgoingBatches(boolean isLastBatch, boolean 
schemaChanged) throws IOException, InterruptedException {
 for (OutgoingRecordBatch batch : outgoingBatches) {
+  if (Thread.interrupted()) {
+throw new InterruptedException();
--- End diff --

Since we are checking for interrupts here already could we remove 
`Thread.currentThread().isInterrupted()` in the flush(boolean schemaChanged) 
method?


> PartitionerDecorator may close partitioners while CustomRunnable are active 
> during query cancellation
> -
>
> Key: DRILL-6295
> URL: https://issues.apache.org/jira/browse/DRILL-6295
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> During query cancellation, in case 
> {{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the 
> {{latch}}), the wait will be interrupted and {{Futures}} cancelled, but there 
> is no guarantee that all {{CustomRunnable}} terminate before returning from 
> {{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and 
> outgoing batches are cleared, leading to clearing of underlying {{Vectors}} 
> and {{DrillBufs}}. This eventually causes unallocated memory access and JVM 
> crash as {{CustomRunnable}} may execute after income/outgoing batches are 
> cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6322) Lateral Join: Common changes - Add new iterOutcome, Operator types, MockRecordBatch for testing

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-6322:
-
Reviewer: Parth Chandra

> Lateral Join: Common changes - Add new iterOutcome, Operator types, 
> MockRecordBatch for testing
> ---
>
> Key: DRILL-6322
> URL: https://issues.apache.org/jira/browse/DRILL-6322
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Add new iterOutcome (EMIT), Operator types (LATERAL and UNNEST), and  
> MockRecordBatch for testing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6333) Generate and host Javadocs on Apache Drill website

[Kunal Khatua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Khatua updated DRILL-6333:

Fix Version/s: 1.14.0

> Generate and host Javadocs on Apache Drill website
> --
>
> Key: DRILL-6333
> URL: https://issues.apache.org/jira/browse/DRILL-6333
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Kunal Khatua
>Priority: Minor
> Fix For: 1.14.0
>
>
> Currently, there is no hosted location of Javadocs for Apache Drill.
> At a minimum, there should be a cleanly generated Javadoc published for each 
> release, along with any Javadoc additions/enhancements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6333) Generate and host Javadocs on Apache Drill website

[Kunal Khatua (JIRA)]

Kunal Khatua created DRILL-6333:
---

 Summary: Generate and host Javadocs on Apache Drill website
 Key: DRILL-6333
 URL: https://issues.apache.org/jira/browse/DRILL-6333
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Kunal Khatua


Currently, there is no hosted location of Javadocs for Apache Drill.

At a minimum, there should be a cleanly generated Javadoc published for each 
release, along with any Javadoc additions/enhancements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6322) Lateral Join: Common changes - Add new iterOutcome, Operator types, MockRecordBatch for testing

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia reassigned DRILL-6322:


Assignee: Sorabh Hamirwasia

> Lateral Join: Common changes - Add new iterOutcome, Operator types, 
> MockRecordBatch for testing
> ---
>
> Key: DRILL-6322
> URL: https://issues.apache.org/jira/browse/DRILL-6322
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Add new iterOutcome (EMIT), Operator types (LATERAL and UNNEST), and  
> MockRecordBatch for testing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6322) Lateral Join: Common changes - Add new iterOutcome, Operator types, MockRecordBatch for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440127#comment-16440127
 ] 

ASF GitHub Bot commented on DRILL-6322:
---

GitHub user sohami opened a pull request:

https://github.com/apache/drill/pull/1211

DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, Operator 
types, MockRecordBatch for testing

@parthchandra - Please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sohami/drill DRILL-6322

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1211.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1211


commit 2dbc6768857a0b8cd3e6e8c19fc133af2abf20d3
Author: Sorabh Hamirwasia 
Date:   2018-02-05T21:12:15Z

DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, 
Operatortypes, MockRecordBatch for testing
Note: Added new Iterator State EMIT, added operatos LATERA_JOIN 
& UNNEST in CoreOperatorType and added LateralContract interface

commit 15655ab3543521ff56f4d426ebf7ed4884eb3006
Author: Sorabh Hamirwasia 
Date:   2018-02-07T21:29:28Z

DRILL-6322: Lateral Join: Common changes - Add new iterOutcome, 
Operatortypes, MockRecordBatch for testing
Note: Implementation of MockRecordBatch to test operator 
behavior for different IterOutcomes.
  a) Creates new output container for schema change cases.
  b) Doesn't create new container for each next() call 
without schema change, since the operator in test expects the ValueVector 
object in it's
 incoming batch to be same unless a OK_NEW_SCHEMA case 
is hit. Since setup() method of operator in test will store the reference to 
value vector received in first batch




> Lateral Join: Common changes - Add new iterOutcome, Operator types, 
> MockRecordBatch for testing
> ---
>
> Key: DRILL-6322
> URL: https://issues.apache.org/jira/browse/DRILL-6322
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Add new iterOutcome (EMIT), Operator types (LATERAL and UNNEST), and  
> MockRecordBatch for testing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[Bridget Bevens (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440036#comment-16440036
 ] 

Bridget Bevens commented on DRILL-6174:
---

Documentation updated to reflect changes described in this JIRA: 
[https://drill.apache.org/docs/parquet-filter-pushdown/#support] 

> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6323) Lateral Join - Initial implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440134#comment-16440134
 ] 

ASF GitHub Bot commented on DRILL-6323:
---

GitHub user sohami opened a pull request:

https://github.com/apache/drill/pull/1212

DRILL-6323: Lateral Join - Initial implementation

@parthchandra - Please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sohami/drill DRILL-6323

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1212


commit 38b82ce8ebca3b91ba8847229a568daf88f654a3
Author: Sorabh Hamirwasia 
Date:   2018-03-06T05:09:16Z

DRILL-6323: Lateral Join - Initial implementation
Refactor Join PopConfigs

commit 0931ac974f7be41cd5ca4516524d04c4e5c6245d
Author: Sorabh Hamirwasia 
Date:   2018-02-05T22:46:19Z

DRILL-6323: Lateral Join - Initial implementation

commit 08b7d38e6ffe4fed492a2ea67c3907f0f514717e
Author: Sorabh Hamirwasia 
Date:   2018-02-20T22:47:48Z

DRILL-6323: Lateral Join - Initial implementation
Note: Refactor, fix various issues with LATERAL: a)Override 
prefetch call in BuildSchema phase for LATERAL, b)EMIT handling in buildSchema, 
c)Issue when in multilevel Lateral case
schema change is observed only on right side of UNNEST, 
d)Handle SelectionVector in incoming batch, e) Handling Schema change, f) 
Updating joinIndexes correctly when producing multiple output batches
for current left inputs.
Added tests for a)EMIT handling in buildSchema, b)Multiple 
UNNEST at same level, c)Multilevel Lateral, d)Multilevel Lateral with Schema 
change on left/right or both branches, e) Left LATERAL join
f)Schema change for UNNEST and Non-UNNEST columns, g)Error 
outcomes from left, h) Producing multiple output batches for given 
incoming, i) Consuming multiple incoming into single output batch

commit f686877198857a23227031329b2f48c163f85021
Author: Sorabh Hamirwasia 
Date:   2018-03-09T23:56:22Z

DRILL-6323: Lateral Join - Initial implementation
Note: Remove codegen and operator template class. Logic to copy 
data is moved to LateralJoinBatch itself

commit cb47407a8f47e7fc10ac04c438f7e903de92a80c
Author: Sorabh Hamirwasia 
Date:   2018-03-13T23:41:03Z

DRILL-6323: Lateral Join - Initial implementation
Note: Add some debug logs for LateralJoinBatch

commit 2ea968e3302ad8bb6d62a8032bd6b6329b900d3a
Author: Sorabh Hamirwasia 
Date:   2018-03-14T23:59:29Z

DRILL-6323: Lateral Join - Initial implementation
Note: Refactor BatchMemorySize to put outputBatchSize in 
abstract class. Created a new JoinBatchMemoryManager to be shared across join 
record batches.
  Changed merge join to use AbstractBinaryRecordBatch instead 
of AbstractRecordBatch, and use JoinBatchMemoryManager

commit fb521c65f91b69543ae9b5dbb9a727c79f852573
Author: Sorabh Hamirwasia 
Date:   2018-03-19T19:00:22Z

DRILL-6323: Lateral Join - Initial implementation
Note: Lateral Join Batch Memory manager support using the 
record batch sizer




> Lateral Join - Initial implementation
> -
>
> Key: DRILL-6323
> URL: https://issues.apache.org/jira/browse/DRILL-6323
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Priority: Major
>
> Implementation of Lateral Join with unit tests using MockRecordBatch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6323) Lateral Join - Initial implementation

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia updated DRILL-6323:
-
Reviewer: Parth Chandra

> Lateral Join - Initial implementation
> -
>
> Key: DRILL-6323
> URL: https://issues.apache.org/jira/browse/DRILL-6323
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Implementation of Lateral Join with unit tests using MockRecordBatch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6323) Lateral Join - Initial implementation

[Sorabh Hamirwasia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sorabh Hamirwasia reassigned DRILL-6323:


Assignee: Sorabh Hamirwasia

> Lateral Join - Initial implementation
> -
>
> Key: DRILL-6323
> URL: https://issues.apache.org/jira/browse/DRILL-6323
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Implementation of Lateral Join with unit tests using MockRecordBatch



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6277) Query fails with DATA_READ ERROR when correlated subquery has "always false" filter

[Volodymyr Vysotskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi resolved DRILL-6277.

Resolution: Fixed

Fixed in the scope of DRILL-6294

> Query fails with DATA_READ ERROR when correlated subquery has "always false" 
> filter
> ---
>
> Key: DRILL-6277
> URL: https://issues.apache.org/jira/browse/DRILL-6277
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.14.0
>
>
> Query with correlated subquery which contains "always false" filter fails:
> {noformat}
> select * from cp.`employee.json` t where exists(select employee_id from 
> cp.`employee.json` where t.employee_id = 3 and 1 = 5);
> Error: DATA_READ ERROR: The top level of your document must either be a 
> single array of maps or a set of white space delimited maps.
> Line  -1
> Column  0
> Field  
> Fragment 0:0
> [Error Id: 66b38c7e-7d12-4f38-93e4-f97f08f55e93 on user515050-pc:31013] 
> (state=,code=0)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6320) Don't Allow Javadoc comments for license headers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440368#comment-16440368
 ] 

ASF GitHub Bot commented on DRILL-6320:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/1207
  
@arina-ielchiieva or @Ben-Zvi would it make sense to merge this in earlier 
before the batch commits later in the week? the number of files changes are 
large so don't want @ilooner to have to rebase each time.


> Don't Allow Javadoc comments for license headers
> 
>
> Key: DRILL-6320
> URL: https://issues.apache.org/jira/browse/DRILL-6320
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Currently some license headers are in a javadoc comment instead of a normal 
> comment. This is not good since we don't want to include spurious license 
> headers in java docs when we publish them. The fix would be to change the rat 
> plugin configuration to not allow java doc comments.
>  
> *For documentation*
> Min required maven version has changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6322) Lateral Join: Common changes - Add new iterOutcome, Operator types, MockRecordBatch for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440166#comment-16440166
 ] 

ASF GitHub Bot commented on DRILL-6322:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/1211
  
+1


> Lateral Join: Common changes - Add new iterOutcome, Operator types, 
> MockRecordBatch for testing
> ---
>
> Key: DRILL-6322
> URL: https://issues.apache.org/jira/browse/DRILL-6322
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Add new iterOutcome (EMIT), Operator types (LATERAL and UNNEST), and  
> MockRecordBatch for testing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440221#comment-16440221
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181927070
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -262,68 +280,122 @@ public FlushBatchesHandlingClass(boolean 
isLastBatch, boolean schemaChanged) {
 }
 
 @Override
-public void execute(Partitioner part) throws IOException {
+public void execute(Partitioner part) throws IOException, 
InterruptedException {
   part.flushOutgoingBatches(isLastBatch, schemaChanged);
 }
   }
 
   /**
-   * Helper class to wrap Runnable with customized naming
-   * Exception handling
+   * Helper class to wrap Runnable with cancellation and waiting for 
completion support
*
*/
-  private static class CustomRunnable implements Runnable {
+  private static class PartitionerTask implements Runnable {
+
+private enum STATE {
+  NEW,
+  COMPLETING,
+  NORMAL,
+  EXCEPTIONAL,
+  CANCELLED,
+  INTERRUPTING,
+  INTERRUPTED
+}
+
+private final AtomicReference state;
+private final AtomicReference runner;
+private final PartitionerDecorator partitionerDecorator;
+private final AtomicInteger count;
 
-private final String parentThreadName;
-private final CountDownLatch latch;
 private final GeneralExecuteIface iface;
-private final Partitioner part;
+private final Partitioner partitioner;
 private CountDownLatchInjection testCountDownLatch;
 
-private volatile IOException exp;
+private volatile ExecutionException exception;
 
-public CustomRunnable(final String parentThreadName, final 
CountDownLatch latch, final GeneralExecuteIface iface,
-final Partitioner part, CountDownLatchInjection 
testCountDownLatch) {
-  this.parentThreadName = parentThreadName;
-  this.latch = latch;
+public PartitionerTask(PartitionerDecorator partitionerDecorator, 
GeneralExecuteIface iface, Partitioner partitioner, AtomicInteger count, 
CountDownLatchInjection testCountDownLatch) {
+  state = new AtomicReference<>(STATE.NEW);
+  runner = new AtomicReference<>();
+  this.partitionerDecorator = partitionerDecorator;
   this.iface = iface;
-  this.part = part;
+  this.partitioner = partitioner;
+  this.count = count;
   this.testCountDownLatch = testCountDownLatch;
 }
 
 @Override
 public void run() {
-  // Test only - Pause until interrupted by fragment thread
-  try {
-testCountDownLatch.await();
-  } catch (final InterruptedException e) {
-logger.debug("Test only: partitioner thread is interrupted in test 
countdown latch await()", e);
-  }
-
-  final Thread currThread = Thread.currentThread();
-  final String currThreadName = currThread.getName();
-  final OperatorStats localStats = part.getStats();
-  try {
-final String newThreadName = parentThreadName + currThread.getId();
-currThread.setName(newThreadName);
+  final Thread thread = Thread.currentThread();
+  if (runner.compareAndSet(null, thread)) {
+final String name = thread.getName();
+thread.setName(String.format("Partitioner-%s-%d", 
partitionerDecorator.thread.getName(), thread.getId()));
+final OperatorStats localStats = partitioner.getStats();
 localStats.clear();
 localStats.startProcessing();
-iface.execute(part);
-  } catch (IOException e) {
-exp = e;
-  } finally {
-localStats.stopProcessing();
-currThread.setName(currThreadName);
-latch.countDown();
+ExecutionException executionException = null;
+try {
+  // Test only - Pause until interrupted by fragment thread
+  testCountDownLatch.await();
+  if (state.get() == STATE.NEW) {
+iface.execute(partitioner);
+  }
+} catch (InterruptedException e) {
+  if (state.compareAndSet(STATE.NEW, STATE.INTERRUPTED)) {
+logger.warn("Partitioner Task interrupted during the run", e);
+  }
+} catch (Throwable t) {
+  executionException = new ExecutionException(t);
+} finally {
+  if (state.compareAndSet(STATE.NEW, STATE.COMPLETING)) {
+ 

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440163#comment-16440163
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181915654
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 ---
@@ -262,68 +280,122 @@ public FlushBatchesHandlingClass(boolean 
isLastBatch, boolean schemaChanged) {
 }
 
 @Override
-public void execute(Partitioner part) throws IOException {
+public void execute(Partitioner part) throws IOException, 
InterruptedException {
   part.flushOutgoingBatches(isLastBatch, schemaChanged);
 }
   }
 
   /**
-   * Helper class to wrap Runnable with customized naming
-   * Exception handling
+   * Helper class to wrap Runnable with cancellation and waiting for 
completion support
*
*/
-  private static class CustomRunnable implements Runnable {
+  private static class PartitionerTask implements Runnable {
+
+private enum STATE {
+  NEW,
+  COMPLETING,
+  NORMAL,
+  EXCEPTIONAL,
+  CANCELLED,
+  INTERRUPTING,
+  INTERRUPTED
+}
+
+private final AtomicReference state;
+private final AtomicReference runner;
+private final PartitionerDecorator partitionerDecorator;
+private final AtomicInteger count;
 
-private final String parentThreadName;
-private final CountDownLatch latch;
 private final GeneralExecuteIface iface;
-private final Partitioner part;
+private final Partitioner partitioner;
 private CountDownLatchInjection testCountDownLatch;
 
-private volatile IOException exp;
+private volatile ExecutionException exception;
 
-public CustomRunnable(final String parentThreadName, final 
CountDownLatch latch, final GeneralExecuteIface iface,
-final Partitioner part, CountDownLatchInjection 
testCountDownLatch) {
-  this.parentThreadName = parentThreadName;
-  this.latch = latch;
+public PartitionerTask(PartitionerDecorator partitionerDecorator, 
GeneralExecuteIface iface, Partitioner partitioner, AtomicInteger count, 
CountDownLatchInjection testCountDownLatch) {
+  state = new AtomicReference<>(STATE.NEW);
+  runner = new AtomicReference<>();
+  this.partitionerDecorator = partitionerDecorator;
   this.iface = iface;
-  this.part = part;
+  this.partitioner = partitioner;
+  this.count = count;
   this.testCountDownLatch = testCountDownLatch;
 }
 
 @Override
 public void run() {
-  // Test only - Pause until interrupted by fragment thread
-  try {
-testCountDownLatch.await();
-  } catch (final InterruptedException e) {
-logger.debug("Test only: partitioner thread is interrupted in test 
countdown latch await()", e);
-  }
-
-  final Thread currThread = Thread.currentThread();
-  final String currThreadName = currThread.getName();
-  final OperatorStats localStats = part.getStats();
-  try {
-final String newThreadName = parentThreadName + currThread.getId();
-currThread.setName(newThreadName);
+  final Thread thread = Thread.currentThread();
+  if (runner.compareAndSet(null, thread)) {
+final String name = thread.getName();
+thread.setName(String.format("Partitioner-%s-%d", 
partitionerDecorator.thread.getName(), thread.getId()));
+final OperatorStats localStats = partitioner.getStats();
 localStats.clear();
 localStats.startProcessing();
-iface.execute(part);
-  } catch (IOException e) {
-exp = e;
-  } finally {
-localStats.stopProcessing();
-currThread.setName(currThreadName);
-latch.countDown();
+ExecutionException executionException = null;
+try {
+  // Test only - Pause until interrupted by fragment thread
+  testCountDownLatch.await();
+  if (state.get() == STATE.NEW) {
+iface.execute(partitioner);
+  }
+} catch (InterruptedException e) {
+  if (state.compareAndSet(STATE.NEW, STATE.INTERRUPTED)) {
+logger.warn("Partitioner Task interrupted during the run", e);
+  }
+} catch (Throwable t) {
+  executionException = new ExecutionException(t);
+} finally {
+  if (state.compareAndSet(STATE.NEW, STATE.COMPLETING)) {
+

[jira] [Commented] (DRILL-6322) Lateral Join: Common changes - Add new iterOutcome, Operator types, MockRecordBatch for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440193#comment-16440193
 ] 

ASF GitHub Bot commented on DRILL-6322:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/1211


> Lateral Join: Common changes - Add new iterOutcome, Operator types, 
> MockRecordBatch for testing
> ---
>
> Key: DRILL-6322
> URL: https://issues.apache.org/jira/browse/DRILL-6322
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Sorabh Hamirwasia
>Priority: Major
>
> Add new iterOutcome (EMIT), Operator types (LATERAL and UNNEST), and  
> MockRecordBatch for testing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6295) PartitionerDecorator may close partitioners while CustomRunnable are active during query cancellation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440219#comment-16440219
 ] 

ASF GitHub Bot commented on DRILL-6295:
---

Github user vrozov commented on a diff in the pull request:

https://github.com/apache/drill/pull/1208#discussion_r181926928
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
 ---
@@ -161,8 +161,11 @@ public OperatorStats getStats() {
* @param schemaChanged  true if the schema has changed
*/
   @Override
-  public void flushOutgoingBatches(boolean isLastBatch, boolean 
schemaChanged) throws IOException {
+  public void flushOutgoingBatches(boolean isLastBatch, boolean 
schemaChanged) throws IOException, InterruptedException {
 for (OutgoingRecordBatch batch : outgoingBatches) {
+  if (Thread.interrupted()) {
+throw new InterruptedException();
--- End diff --

I'll revert back throwing 'InterruptedException' to avoid mishandling of 
the last batch. 


> PartitionerDecorator may close partitioners while CustomRunnable are active 
> during query cancellation
> -
>
> Key: DRILL-6295
> URL: https://issues.apache.org/jira/browse/DRILL-6295
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.14.0
>
>
> During query cancellation, in case 
> {{PartitionerDecorator.executeMethodLogic()}} is active (waiting on the 
> {{latch}}), the wait will be interrupted and {{Futures}} cancelled, but there 
> is no guarantee that all {{CustomRunnable}} terminate before returning from 
> {{PartitionerDecorator.executeMethodLogic()}}. On exit, both income and 
> outgoing batches are cleared, leading to clearing of underlying {{Vectors}} 
> and {{DrillBufs}}. This eventually causes unallocated memory access and JVM 
> crash as {{CustomRunnable}} may execute after income/outgoing batches are 
> cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6334) Code cleanup

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6334:
--

 Summary: Code cleanup
 Key: DRILL-6334
 URL: https://issues.apache.org/jira/browse/DRILL-6334
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.14.0


Minor cleanup of a few files. Done as a separate PR to allow easier review.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-4807) ORDER BY aggregate function in window definition results in AssertionError: Internal error: invariant violated: conversion result not null

[Volodymyr Vysotskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi resolved DRILL-4807.

Resolution: Fixed

Fixed in the scope of DRILL-6294

> ORDER BY aggregate function in window definition results in AssertionError: 
> Internal error: invariant violated: conversion result not null
> --
>
> Key: DRILL-4807
> URL: https://issues.apache.org/jira/browse/DRILL-4807
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0, 1.10.0
>Reporter: Khurram Faraaz
>Assignee: Volodymyr Tkach
>Priority: Major
>  Labels: window_function
>
> This seems to be a problem when regular window function queries, when 
> aggregate function is used in ORDER BY clause inside the window definition.
> MapR Drill 1.8.0 commit ID : 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT col0, SUM(col0) OVER ( PARTITION BY col7 
> ORDER BY MIN(col8)) avg_col0, col7 FROM `allTypsUniq.parquet` GROUP BY 
> col0,col8,col7;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated: 
> conversion result not null
> [Error Id: 19a3eced--4e83-ae0f-6b8ea21b2afd on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT col0, AVG(col0) OVER ( PARTITION BY col7 
> ORDER BY MIN(col8)) avg_col0, col7 FROM `allTypsUniq.parquet` GROUP BY 
> col0,col8,col7;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated: 
> conversion result not null
> [Error Id: c9b7ebf2-6097-41d8-bb73-d57da4ace8ad on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-07-26 09:26:16,717 [2868d347-3124-0c58-89ff-19e4ee891031:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 2868d347-3124-0c58-89ff-19e4ee891031: SELECT col0, AVG(col0) OVER ( PARTITION 
> BY col7 ORDER BY MIN(col8)) avg_col0, col7 FROM `allTypsUniq.parquet` GROUP 
> BY col0,col8,col7
> 2016-07-26 09:26:16,751 [2868d347-3124-0c58-89ff-19e4ee891031:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: Internal 
> error: invariant violated: conversion result not null
> [Error Id: c9b7ebf2-6097-41d8-bb73-d57da4ace8ad on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> AssertionError: Internal error: invariant violated: conversion result not null
> [Error Id: c9b7ebf2-6097-41d8-bb73-d57da4ace8ad on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected 
> exception during fragment initialization: Internal error: invariant violated: 
> conversion result not null
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: invariant violated: 
> conversion result not null
> at org.apache.calcite.util.Util.newInternal(Util.java:777) 
> ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at org.apache.calcite.util.Util.permAssert(Util.java:885) 
> ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4063)
>  ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertSortExpression(SqlToRelConverter.java:4080)
>  ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1783)
>  ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1100(SqlToRelConverter.java:185)
>  ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> 

[jira] [Commented] (DRILL-6301) Parquet Performance Analysis

[Vlad Rozov (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440280#comment-16440280
 ] 

Vlad Rozov commented on DRILL-6301:
---

[~sachouche] I don't understand the usage of "text transformation where every 
byte is replaced by its next byte" for the benchmark. Is it the most common 
transformation done while processing parquet files?

> Parquet Performance Analysis
> 
>
> Key: DRILL-6301
> URL: https://issues.apache.org/jira/browse/DRILL-6301
> Project: Apache Drill
>  Issue Type: Task
>  Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> _*Description -*_
>  * DRILL-5846 is meant to improve the Flat Parquet reader performance
>  * The associated implementation resulted in a 2x - 4x performance improvement
>  * Though during the review process ([pull 
> request|[https://github.com/apache/drill/pull/1060])] few key questions arised
>  
> *_Intermediary Processing via Direct Memory vs Byte Arrays_*
>  * The main reasons for using byte arrays for intermediary processing is to 
> a) avoid the high cost of the DrillBuf checks (especially the reference 
> counting) and b) benefit from some observed Java optimizations when accessing 
> byte arrays
>  * Starting with version 1.12.0, the DrillBuf enablement checks have been 
> refined so that memory access and reference counting checks can be enabled 
> independently
>  * Benchmarking of Java's Direct Memory unsafe method using JMH indicates the 
> performance gap between heap vs direct memory  is very narrow except for few 
> use-cases
>  * There are also concerns that the extra copy step (from direct memory into 
> byte arrays) will have a negative effect on performance; note that this 
> overhead was not observed using Intel's Vtune as the intermediary buffer were 
> a) pinned to a single CPU, b) reused, and c) small enough to remain in the L1 
> cache during columnar processing.
> _*Goal*_ 
>  * The Flat Parquet reader is amongst the few Drill columnar operators
>  * It is imperative that we agree on the most optimal processing pattern so 
> that the decisions that we take within this Jira are not only applied to 
> Parquet but to all Drill columnar operators   
> _*Methodology*_ 
>  # Assess the performance impact of using intermediary byte arrays (as 
> described above)
>  # Prototype a solution using Direct Memory and DrillBuf checks off, access 
> checks on, all checks on
>  # Make an educated decision on which processing pattern should be adopted
>  # Decide whether it is ok to use Java's unsafe API (and through what 
> mechanism) on byte arrays (when the use of byte arrays is a necessity)
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6335) Refactor row set abstractions to prepare for unions

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6335:
--

 Summary: Refactor row set abstractions to prepare for unions
 Key: DRILL-6335
 URL: https://issues.apache.org/jira/browse/DRILL-6335
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.14.0


The row set abstractions will eventually support unions and lists. The changes 
to support these types are extensive. This PR introduces refactoring that puts 
the pieces in the correct location to allow for inserting those additional 
types.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)