[jira] [Created] (DRILL-4840) Sqlline prints log output to stdout on startup

2016-08-09 Thread Parth Chandra (JIRA) 
Parth Chandra created DRILL-4840:


 Summary: Sqlline prints log output to stdout on startup
 Key: DRILL-4840
 URL: https://issues.apache.org/jira/browse/DRILL-4840
 Project: Apache Drill
  Issue Type: Bug
Reporter: Parth Chandra


With the refactoring of drill scripts, sqlline has now started printing logging 
messages to stdout when it starts up. This messes up some users' scripts that 
invoke sqlline.

See also, DRILL-2798 which was logged because end users had scripts that broke 
as a result of printing out additional information.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4792) Include session options used for a query as part of the profile

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4792:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> Include session options used for a query as part of the profile
> ---
>
> Key: DRILL-4792
> URL: https://issues.apache.org/jira/browse/DRILL-4792
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: no_session_options.JPG, session_options_block.JPG, 
> session_options_collapsed.JPG, session_options_json.JPG
>
>
> Include session options used for a query as part of the profile.
> This will be very useful for debugging/diagnostics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4560) ZKClusterCoordinator does not call DrillbitStatusListener.drillbitRegistered for new bits

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4560:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> ZKClusterCoordinator does not call DrillbitStatusListener.drillbitRegistered 
> for new bits
> -
>
> Key: DRILL-4560
> URL: https://issues.apache.org/jira/browse/DRILL-4560
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components:  Server
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> ZKClusterCoordinator notifies listeners of type DrillbitStatusListener when 
> drillbits disappear from ZooKeeper. The YARN Application Master (AM) also 
> needs to know when bits register themselves with ZK. So, ZKClusterCoordinator 
> should change to detect new Drill-bits, then call 
> DrillbitStatusListener.drillbitRegistered with the new Drill-bits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4714) Add metadata and prepared statement APIs to DrillClient<->Drillbit interface

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4714:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> Add metadata and prepared statement APIs to DrillClient<->Drillbit interface
> 
>
> Key: DRILL-4714
> URL: https://issues.apache.org/jira/browse/DRILL-4714
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Metadata
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.9.0
>
>
> Currently ODBC/JDBC drivers spawn a set of queries on INFORMATION_SCHEMA for 
> metadata. Client has to deal with submitting a query, reading query results 
> and constructing required objects. Sometimes same work is done twice 
> (planning work in case of prepare statements) to get the metadata and execute 
> query. Instead we could simplify the client by providing APIs on the client 
> interface and let the server construct the required objects and send them to 
> client directly. These APIs provide common info that can be consumed by the 
> JDBC/ODBC driver.
> [Here|https://docs.google.com/document/d/1ytwGDYGbEFuS3_eUEqRikh7wQABFyHjS0fXJ6wr1eSU/edit?usp=sharing]
>  is the doc explaining the new APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4752) Remove submit_plan script from Drill distribution

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4752:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> Remove submit_plan script from Drill distribution
> -
>
> Key: DRILL-4752
> URL: https://issues.apache.org/jira/browse/DRILL-4752
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> The Drill distribution includes a submit_plan script in $DRILL_HOME/bin. In 
> Drill's early days, before Drill has a SQL parser and planner, users 
> submitted query plans directly to Drill. Since that time, the other Drill 
> scripts have evolved to support configuration directories, memory options, 
> and more. The old submit_plan script does not include these enhancements.
> It appears that the Drill unit tests do not exercise the script, meaning that 
> we have no way to verify that it still works.
> Rather than invest time to "modernize" submit_plan, we propose to drop it 
> from the distribution. The script will remain in the Git source tree for the 
> odd case where someone needs it (and wants to enhance the script to use 
> drill-config.sh, etc.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4658) cannot specify tab as a fieldDelimiter in table function

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4658:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> cannot specify tab as a fieldDelimiter in table function
> 
>
> Key: DRILL-4658
> URL: https://issues.apache.org/jira/browse/DRILL-4658
> Project: Apache Drill
>  Issue Type: Bug
>  Components: SQL Parser
>Affects Versions: 1.6.0
> Environment: Mac OS X, Java 8
>Reporter: Vince Gonzalez
>Assignee: Arina Ielchiieva
> Fix For: 1.9.0
>
>
> I can't specify a tab delimiter in the table function because it maybe counts 
> the characters rather than trying to interpret as a character escape code?
> {code}
> 0: jdbc:drill:zk=local> select columns[0] as a, cast(columns[1] as bigint) as 
> b from table(dfs.tmp.`sample_cast.tsv`(type => 'text', fieldDelimiter => 
> '\t', skipFirstLine => true));
> Error: PARSE ERROR: Expected single character but was String: \t
> table sample_cast.tsv
> parameter fieldDelimiter
> SQL Query null
> [Error Id: 3efa82e1-3810-4d4a-b23c-32d6658dffcf on 172.30.1.144:31010] 
> (state=,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3726) Drill is not properly interpreting CRLF (0d0a). CR gets read as content.

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-3726:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> Drill is not properly interpreting CRLF (0d0a). CR gets read as content.
> 
>
> Key: DRILL-3726
> URL: https://issues.apache.org/jira/browse/DRILL-3726
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Text & CSV
>Affects Versions: 1.1.0
> Environment: Linux RHEL 6.6, OSX 10.9
>Reporter: Edmon Begoli
>Assignee: Arina Ielchiieva
> Fix For: 1.9.0
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
>   When we query the last attribute of a text file, we get missing characters. 
>  Looking at the row through Drill, a \r is included at the end of the last 
> attribute.  
> Looking in a text editor, it's not embedded into that attribute.
> I'm thinking that Drill is not interpreting CRLF (0d0a) as a new line, only 
> the LF, resulting in the CR becoming part of the last attribute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4730) Update JDBC DatabaseMetaData implementation to use new Metadata APIs

2016-08-09 Thread Jinfeng Ni (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jinfeng Ni updated DRILL-4730:
--
Fix Version/s: (was: 1.8.0)
   1.9.0

> Update JDBC DatabaseMetaData implementation to use new Metadata APIs
> 
>
> Key: DRILL-4730
> URL: https://issues.apache.org/jira/browse/DRILL-4730
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Client - JDBC
>Reporter: Venki Korukanti
>Assignee: Venki Korukanti
> Fix For: 1.9.0
>
>
> DRILL-4728 is going to add support for new metadata APIs. Replace the 
> INFORMATION_SCHEMA queries used to get the metadata with the new APIs 
> provided in Java client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4839) NumberFormatException with union all when a column on one side contains all nulls

2016-08-09 Thread Rahul Challapalli (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Challapalli updated DRILL-4839:
-
Attachment: l_3level.tgz
error.log

> NumberFormatException with union all when a column on one side contains all 
> nulls
> -
>
> Key: DRILL-4839
> URL: https://issues.apache.org/jira/browse/DRILL-4839
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.7.0
>Reporter: Rahul Challapalli
> Attachments: error.log, l_3level.tgz
>
>
> git.commit.id.abbrev=ef6e522
> When I do a union all between the below 2 queries, I got a number format 
> exception.
> {code}
> select l_orderkey, dir0 from `l_3level/1/one` where l_discount > 0.05 limit 5;
> +-++
> | l_orderkey  |dir0|
> +-++
> | 1   | 2015-7-13  |
> | 1   | 2015-7-13  |
> | 1   | 2015-7-13  |
> | 1   | 2015-7-13  |
> | 3   | 2015-7-13  |
> +-++
> 5 rows selected (0.313 seconds)
> 0: jdbc:drill:zk=10.10.100.190:5181> select l_orderkey, dir0 from 
> `l_3level/1/one/2015-7-12`;
> +-+---+
> | l_orderkey  | dir0  |
> +-+---+
> | 1   | null  |
> | 1   | null  |
> | 1   | null  |
> | 1   | null  |
> | 1   | null  |
> | 1   | null  |
> | 2   | null  |
> | 3   | null  |
> | 3   | null  |
> | 3   | null  |
> +-+---+
> 10 rows selected (0.25 seconds)
> {code}
> I attached the error log and the data set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4839) NumberFormatException with union all when a column on one side contains all nulls

2016-08-09 Thread Rahul Challapalli (JIRA) 
Rahul Challapalli created DRILL-4839:


 Summary: NumberFormatException with union all when a column on one 
side contains all nulls
 Key: DRILL-4839
 URL: https://issues.apache.org/jira/browse/DRILL-4839
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Relational Operators
Affects Versions: 1.7.0
Reporter: Rahul Challapalli


git.commit.id.abbrev=ef6e522

When I do a union all between the below 2 queries, I got a number format 
exception.

{code}
select l_orderkey, dir0 from `l_3level/1/one` where l_discount > 0.05 limit 5;
+-++
| l_orderkey  |dir0|
+-++
| 1   | 2015-7-13  |
| 1   | 2015-7-13  |
| 1   | 2015-7-13  |
| 1   | 2015-7-13  |
| 3   | 2015-7-13  |
+-++
5 rows selected (0.313 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> select l_orderkey, dir0 from 
`l_3level/1/one/2015-7-12`;
+-+---+
| l_orderkey  | dir0  |
+-+---+
| 1   | null  |
| 1   | null  |
| 1   | null  |
| 1   | null  |
| 1   | null  |
| 1   | null  |
| 2   | null  |
| 3   | null  |
| 3   | null  |
| 3   | null  |
+-+---+
10 rows selected (0.25 seconds)
{code}

I attached the error log and the data set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4836.

Resolution: Fixed

Fixed in 
[0a4c21c|https://github.com/apache/drill/commit/0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1].

> ZK Issue during Drillbit startup, possibly due to race condition
> 
>
> Key: DRILL-4836
> URL: https://issues.apache.org/jira/browse/DRILL-4836
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Reporter: Abhishek Girish
>Assignee: Paul Rogers
> Fix For: 1.8.0
>
>
> During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
> during startup:
> {code}
> Exception in thread "main" 
> org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
> initial startup of Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable 
> to put
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
> at 
> org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
> ...
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
> ... 2 more
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
> ... 7 more
> {code}
> And similarly,
> {code}
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4581) Various problems in the Drill startup scripts

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4581.

   Resolution: Fixed
Fix Version/s: 1.8.0

> Various problems in the Drill startup scripts
> -
>
> Key: DRILL-4581
> URL: https://issues.apache.org/jira/browse/DRILL-4581
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components:  Server
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.8.0
>
>
> Noticed the following in drillbit.sh:
> 1) Comment: DRILL_LOG_DIRWhere log files are stored.  PWD by default.
> Code: DRILL_LOG_DIR=/var/log/drill or, if it does not exist, $DRILL_HOME/log
> 2) Comment: DRILL_PID_DIRThe pid files are stored. /tmp by default.
> Code: DRILL_PID_DIR=$DRILL_HOME
> 3) Redundant checking of JAVA_HOME. drillbit.sh sources drill-config.sh which 
> checks JAVA_HOME. Later, drillbit.sh checks it again. The second check is 
> both unnecessary and prints a less informative message than the 
> drill-config.sh check. Suggestion: Remove the JAVA_HOME check in drillbit.sh.
> 4) Though drill-config.sh carefully checks JAVA_HOME, it does not export the 
> JAVA_HOME variable. Perhaps this is why drillbit.sh repeats the check? 
> Recommended: export JAVA_HOME from drill-config.sh.
> 5) Both drillbit.sh and the sourced drill-config.sh check DRILL_LOG_DIR and 
> set the default value. Drill-config.sh defaults to /var/log/drill, or if that 
> fails, to $DRILL_HOME/log. Drillbit.sh just sets /var/log/drill and does not 
> handle the case where that directory is not writable. Suggested: remove the 
> check in drillbit.sh.
> 6) Drill-config.sh checks the writability of the DRILL_LOG_DIR by touching 
> sqlline.log, but does not delete that file, leaving a bogus, empty client log 
> file on the drillbit server. Recommendation: use bash commands instead.
> 7) The implementation of the above check is a bit awkward. It has a fallback 
> case with somewhat awkward logic. Clean this up.
> 8) drillbit.sh, but not drill-config.sh, attempts to create /var/log/drill if 
> it does not exist. Recommended: decide on a single choice, implement it in 
> drill-config.sh.
> 9) drill-config.sh checks if $DRILL_CONF_DIR is a directory. If not, defaults 
> it to $DRILL_HOME/conf. This can lead to subtle errors. If I use
> drillbit.sh --config /misspelled/path
> where I mistype the path, I won't get an error, I get the default config, 
> which may not at all be what I want to run. Recommendation: if the value of 
> DRILL_CONF_DRILL is passed into the script (as a variable or via --config), 
> then that directory must exist. Else, use the default.
> 10) drill-config.sh exports, but may not set, HADOOP_HOME. This may be left 
> over from the original Hadoop script that the Drill script was based upon. 
> Recomendation: export only in the case that HADOOP_HOME is set for cygwin.
> 11) Drill-config.sh checks JAVA_HOME and prints a big, bold error message to 
> stderr if JAVA_HOME is not set. Then, it checks the Java version and prints a 
> different message (to stdout) if the version is wrong. Recommendation: use 
> the same format (and stderr) for both.
> 12) Similarly, other Java checks later in the script produce messages to 
> stdout, not stderr.
> 13) Drill-config.sh searches $JAVA_HOME to find java/java.exe and verifies 
> that it is executable. The script then throws away what we just found. Then, 
> drill-bit.sh tries to recreate this information as:
> JAVA=$JAVA_HOME/bin/java
> This is wrong in two ways: 1) it ignores the actual java location and assumes 
> it, and 2) it does not handle the java.exe case that drill-config.sh 
> carefully worked out.
> Recommendation: export JAVA from drill-config.sh and remove the above line 
> from drillbit.sh.
> 14) drillbit.sh presumably takes extra arguments like this:
> drillbit.sh -Dvar0=value0 --config /my/conf/dir start -Dvar1=value1 
> -Dvar2=value2 -Dvar3=value3
> The -D bit allows the user to override config variables at the command line. 
> But, the scripts don't use the values.
> A) drill-config.sh consumes --config /my/conf/dir after consuming the leading 
> arguments:
> while [ $# -gt 1 ]; do
>   if [ "--config" = "$1" ]; then
> shift
> confdir=$1
> shift
> DRILL_CONF_DIR=$confdir
>   else
> # Presume we are at end of options and break
> break
>   fi
> done
> B) drill-bit.sh will discard the var1:
> startStopStatus=$1 <-- grabs "start"
> shift
> command=drillbit
> shift   <-- Consumes -Dvar1=value1
> C) Remaining values passed back into drillbit.sh:
> args=$@
> nohup $thiscmd internal_start $command $args
> D) Second invocation discards -Dvar2=value2 as described above.
> E) Remaining values are passed to runbit:
>

[jira] [Reopened] (DRILL-4581) Various problems in the Drill startup scripts

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam reopened DRILL-4581:


> Various problems in the Drill startup scripts
> -
>
> Key: DRILL-4581
> URL: https://issues.apache.org/jira/browse/DRILL-4581
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components:  Server
>Affects Versions: 1.6.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.8.0
>
>
> Noticed the following in drillbit.sh:
> 1) Comment: DRILL_LOG_DIRWhere log files are stored.  PWD by default.
> Code: DRILL_LOG_DIR=/var/log/drill or, if it does not exist, $DRILL_HOME/log
> 2) Comment: DRILL_PID_DIRThe pid files are stored. /tmp by default.
> Code: DRILL_PID_DIR=$DRILL_HOME
> 3) Redundant checking of JAVA_HOME. drillbit.sh sources drill-config.sh which 
> checks JAVA_HOME. Later, drillbit.sh checks it again. The second check is 
> both unnecessary and prints a less informative message than the 
> drill-config.sh check. Suggestion: Remove the JAVA_HOME check in drillbit.sh.
> 4) Though drill-config.sh carefully checks JAVA_HOME, it does not export the 
> JAVA_HOME variable. Perhaps this is why drillbit.sh repeats the check? 
> Recommended: export JAVA_HOME from drill-config.sh.
> 5) Both drillbit.sh and the sourced drill-config.sh check DRILL_LOG_DIR and 
> set the default value. Drill-config.sh defaults to /var/log/drill, or if that 
> fails, to $DRILL_HOME/log. Drillbit.sh just sets /var/log/drill and does not 
> handle the case where that directory is not writable. Suggested: remove the 
> check in drillbit.sh.
> 6) Drill-config.sh checks the writability of the DRILL_LOG_DIR by touching 
> sqlline.log, but does not delete that file, leaving a bogus, empty client log 
> file on the drillbit server. Recommendation: use bash commands instead.
> 7) The implementation of the above check is a bit awkward. It has a fallback 
> case with somewhat awkward logic. Clean this up.
> 8) drillbit.sh, but not drill-config.sh, attempts to create /var/log/drill if 
> it does not exist. Recommended: decide on a single choice, implement it in 
> drill-config.sh.
> 9) drill-config.sh checks if $DRILL_CONF_DIR is a directory. If not, defaults 
> it to $DRILL_HOME/conf. This can lead to subtle errors. If I use
> drillbit.sh --config /misspelled/path
> where I mistype the path, I won't get an error, I get the default config, 
> which may not at all be what I want to run. Recommendation: if the value of 
> DRILL_CONF_DRILL is passed into the script (as a variable or via --config), 
> then that directory must exist. Else, use the default.
> 10) drill-config.sh exports, but may not set, HADOOP_HOME. This may be left 
> over from the original Hadoop script that the Drill script was based upon. 
> Recomendation: export only in the case that HADOOP_HOME is set for cygwin.
> 11) Drill-config.sh checks JAVA_HOME and prints a big, bold error message to 
> stderr if JAVA_HOME is not set. Then, it checks the Java version and prints a 
> different message (to stdout) if the version is wrong. Recommendation: use 
> the same format (and stderr) for both.
> 12) Similarly, other Java checks later in the script produce messages to 
> stdout, not stderr.
> 13) Drill-config.sh searches $JAVA_HOME to find java/java.exe and verifies 
> that it is executable. The script then throws away what we just found. Then, 
> drill-bit.sh tries to recreate this information as:
> JAVA=$JAVA_HOME/bin/java
> This is wrong in two ways: 1) it ignores the actual java location and assumes 
> it, and 2) it does not handle the java.exe case that drill-config.sh 
> carefully worked out.
> Recommendation: export JAVA from drill-config.sh and remove the above line 
> from drillbit.sh.
> 14) drillbit.sh presumably takes extra arguments like this:
> drillbit.sh -Dvar0=value0 --config /my/conf/dir start -Dvar1=value1 
> -Dvar2=value2 -Dvar3=value3
> The -D bit allows the user to override config variables at the command line. 
> But, the scripts don't use the values.
> A) drill-config.sh consumes --config /my/conf/dir after consuming the leading 
> arguments:
> while [ $# -gt 1 ]; do
>   if [ "--config" = "$1" ]; then
> shift
> confdir=$1
> shift
> DRILL_CONF_DIR=$confdir
>   else
> # Presume we are at end of options and break
> break
>   fi
> done
> B) drill-bit.sh will discard the var1:
> startStopStatus=$1 <-- grabs "start"
> shift
> command=drillbit
> shift   <-- Consumes -Dvar1=value1
> C) Remaining values passed back into drillbit.sh:
> args=$@
> nohup $thiscmd internal_start $command $args
> D) Second invocation discards -Dvar2=value2 as described above.
> E) Remaining values are passed to runbit:
> "$DRILL_HOME"/bin/runbit  $command "$@" start
> F) Where they

[jira] [Updated] (DRILL-4623) Disable Epoll by Default

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-4623:
---
Fix Version/s: 1.8.0

> Disable Epoll by Default
> 
>
> Key: DRILL-4623
> URL: https://issues.apache.org/jira/browse/DRILL-4623
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
> Fix For: 1.8.0
>
>
> At higher concurrency (and/or spuriously), we hit [netty issue 
> #3539|https://github.com/netty/netty/issues/3539]. This is an issue with the 
> version of Netty Drill currently uses. Once Drill moves to a later version of 
> Netty, epoll should be reenabled by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4822) Extend distrib-env.sh search to consider site directory

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-4822:
---
Fix Version/s: 1.8.0

> Extend distrib-env.sh search to consider site directory
> ---
>
> Key: DRILL-4822
> URL: https://issues.apache.org/jira/browse/DRILL-4822
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Priority: Minor
> Fix For: 1.8.0
>
>
> DRILL-4581 provided revisions to the Drill launch scripts. As part of that 
> fix, we introduced a new distrib-env.sh file to hold settings created by 
> custom Drill installers (that is, by custom distributions.) The original 
> version of this feature looks for distrib-env.sh only in $DRILL_HOME/env.
> Experience suggests that installers will write site-specific values to 
> distrib-env.sh and so the file must then be copied to $DRILL_SITE when 
> running under YARN. Add $DRILL_SITE to the search path in drill-config.sh for 
> distrib-env.sh.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4499) Remove unused classes

2016-08-09 Thread Sudheesh Katkam (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam updated DRILL-4499:
---
Fix Version/s: 1.8.0

> Remove unused classes
> -
>
> Key: DRILL-4499
> URL: https://issues.apache.org/jira/browse/DRILL-4499
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Sudheesh Katkam
>Assignee: Sudheesh Katkam
> Fix For: 1.8.0
>
>
> List of unused classes that I've tracked over time:
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestPrune.java
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/MethodMap.java
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionBody.java
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/Multitimer.java
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QuerySetup.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/AvailabilityListener.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlCommand.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/SendProgress.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/SendProgress.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/DrillUser.java
> exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordRecorder.java
> exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/PartialWork.java
> exec/java-exec/src/main/java/org/apache/drill/exec/util/AtomicState.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/RecordOutputStream.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/ResourceRequest.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/RootNodeDriver.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414051#comment-15414051
 ] 

ASF GitHub Bot commented on DRILL-4836:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/564


> ZK Issue during Drillbit startup, possibly due to race condition
> 
>
> Key: DRILL-4836
> URL: https://issues.apache.org/jira/browse/DRILL-4836
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Reporter: Abhishek Girish
>Assignee: Paul Rogers
> Fix For: 1.8.0
>
>
> During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
> during startup:
> {code}
> Exception in thread "main" 
> org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
> initial startup of Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable 
> to put
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
> at 
> org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
> ...
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
> ... 2 more
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
> ... 7 more
> {code}
> And similarly,
> {code}
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414052#comment-15414052
 ] 

ASF GitHub Bot commented on DRILL-4766:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/561


> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4796) AssertionError - Nested sum(avg(c1)) over window

2016-08-09 Thread Gautam Kumar Parai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gautam Kumar Parai resolved DRILL-4796.
---
Resolution: Fixed

Closed with commit: 0bac42dec63a46ca787f6c5fe5a51b9a97e0d6cc

> AssertionError - Nested sum(avg(c1)) over window
> 
>
> Key: DRILL-4796
> URL: https://issues.apache.org/jira/browse/DRILL-4796
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.8.0
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
>
> Nested window function query fails on MapR Drill 1.8.0 commit ID 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select sum(avg(c1)) over (partition by c2) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: AssertionError: todo: implement syntax 
> FUNCTION_STAR(COUNT($1))
> [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> stack trace from drillbit.log
> {noformat}
> 2016-07-21 11:25:40,023 [286f4ecc-59bd-113e-1edf-d93411b255aa:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 286f4ecc-59bd-113e-1edf-d93411b255aa: select sum(avg(c1)) over (partition by 
> c2) from `tblWnulls.parquet`
> ...
> 2016-07-21 11:25:40,183 [286f4ecc-59bd-113e-1edf-d93411b255aa:foreman] ERROR 
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: todo: 
> implement syntax FUNCTION_STAR(COUNT($1))
> [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> AssertionError: todo: implement syntax FUNCTION_STAR(COUNT($1))
> [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) 
> [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) 
> [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: 
> Unexpected exception during fragment initialization: todo: implement syntax 
> FUNCTION_STAR(COUNT($1))
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: todo: implement syntax 
> FUNCTION_STAR(COUNT($1))
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:198)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) 
> ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.doFunction(DrillOptiq.java:205)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:105)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) 
> ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14]
> at 
> org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:77) 
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.common.DrillProjectRelBase.getProjectExpressions(DrillProjectRelBase.java:111)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:59)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.physical.SortPrel.getPhysicalOperator(SortPrel.java:81)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.planner.physical.SelectionVectorRemoverPrel.getPhysicalOperator(SelectionVectorRemoverPrel.java:48)
>

[jira] [Resolved] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException

2016-08-09 Thread Gautam Kumar Parai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gautam Kumar Parai resolved DRILL-4795.
---
Resolution: Fixed

Closed with commit: 0bac42dec63a46ca787f6c5fe5a51b9a97e0d6cc

> Nested aggregate windowed query fails - IllegalStateException 
> --
>
> Key: DRILL-4795
> URL: https://issues.apache.org/jira/browse/DRILL-4795
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
>Priority: Critical
> Attachments: tblWnulls.parquet
>
>
> The below two window function queries fail on MapR Drill 1.8.0 commit ID 
> 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> From drillbit.log
> {noformat}
> 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by 
> c2) from `tblWnulls.parquet`
> ...
> 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: java.lang.IllegalStateException: This generator does not support 
> mappings beyond
> at 
> org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) 
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
>  ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
>

[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413971#comment-15413971
 ] 

ASF GitHub Bot commented on DRILL-4836:
---

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/564
  
+1


> ZK Issue during Drillbit startup, possibly due to race condition
> 
>
> Key: DRILL-4836
> URL: https://issues.apache.org/jira/browse/DRILL-4836
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Reporter: Abhishek Girish
>Assignee: Paul Rogers
> Fix For: 1.8.0
>
>
> During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
> during startup:
> {code}
> Exception in thread "main" 
> org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
> initial startup of Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable 
> to put
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
> at 
> org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
> ...
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
> ... 2 more
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
> ... 7 more
> {code}
> And similarly,
> {code}
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413944#comment-15413944
 ] 

ASF GitHub Bot commented on DRILL-4836:
---

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/564

DRILL-4836: ZK Issue during Drillbit startup, possibly due to race condition

ZK Issue during Drillbit startup, possibly due to race condition.
A change made in February created a race condition if two Drillbits
attempt to create the same storage plugin node at the same time.
Revised the code to eliminate the race condition by relying on an
exception to detect that the node already exists.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-rogers/drill DRILL-4836

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/564.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #564


commit 14d8d0e179419e86d86250eae0b7ae0254162ea7
Author: Paul Rogers 
Date:   2016-08-09T03:16:59Z

DRILL-4836

ZK Issue during Drillbit startup, possibly due to race condition.
A change made in February created a race condition if two Drillbits
attempt to create the same storage plugin node at the same time.
Revised the code to eliminate the race condition by relying on an
exception to detect that the node already exists.




> ZK Issue during Drillbit startup, possibly due to race condition
> 
>
> Key: DRILL-4836
> URL: https://issues.apache.org/jira/browse/DRILL-4836
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Reporter: Abhishek Girish
>Assignee: Paul Rogers
> Fix For: 1.8.0
>
>
> During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
> during startup:
> {code}
> Exception in thread "main" 
> org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
> initial startup of Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable 
> to put
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
> at 
> org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
> ...
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
> ... 2 more
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
> ... 7 more
> {code}
> And similarly,
> {code}
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4802) NULLS are not first when NULLS FIRST is used with ORDER BY in window definition

2016-08-09 Thread Khurram Faraaz (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Khurram Faraaz updated DRILL-4802:
--
Assignee: (was: Khurram Faraaz)

> NULLS are not first when NULLS FIRST is used with ORDER BY in window 
> definition
> ---
>
> Key: DRILL-4802
> URL: https://issues.apache.org/jira/browse/DRILL-4802
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>
> NULLS FIRST is not honored when used with ORDER BY inside window definition. 
> This in a wrong results issue.
> MapR Drill 1.8.0 commit ID : 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select c2, AVG(SUM(c1)) OVER(partition by c2 
> order by c2 nulls first) FROM `tblWnulls.parquet` group by c2;
> +---++
> |  c2   | EXPR$1 |
> +---++
> | a | 11152.0|
> | b | 41.0   |
> | c | 56.0   |
> | d | 4.294967315E9  |
> | e | 14.0   |
> | null  | 106.0  |
> +---++
> 6 rows selected (0.227 seconds)
> {noformat}
> {noformat}
> postgres=# select c2, AVG(SUM(c1)) OVER(partition by c2 order by c2 nulls 
> first) FROM t222 group by c2;
>  c2 |  avg   
> +
> |   106.
>  a  | 11152.
>  b  |41.
>  c  |56.
>  d  |4294967315.
>  e  |14.
> (6 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-2330) Add support for nested aggregate expressions for window aggregates

2016-08-09 Thread Dechang Gu (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413801#comment-15413801
 ] 

Dechang Gu commented on DRILL-2330:
---

The Queries are added to the Drill Perf Test Framework, and test on 1.8.0 
master shows the following results:
Env:  UCS micro cluster MapR 5.1.0.37549.GA, TPCDS SF100 parquet files, Apache 
Drill 1.8.0 Master (gitid aaf220f)
   Query time (ms)  
tpcds query   Run 1 Run 2   Run 3
12  DRILL-4525 Validataion Error

20  DRILL-4525 Validataion Error

47  65,515  27,391  26,247
51  DRILL-4811 IllegalArgument  
53  7,722   4,310   3,518
57 22,968   15,956  15,578
63 7,2924,012   3,526
89 5,5923,777   3,585
98DRILL-4525 Validataion Error  

> Add support for nested aggregate expressions for window aggregates
> --
>
> Key: DRILL-2330
> URL: https://issues.apache.org/jira/browse/DRILL-2330
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Abhishek Girish
>Assignee: Gautam Kumar Parai
> Fix For: 1.8.0
>
> Attachments: drillbit.log
>
>
> Aggregate expressions currently cannot be nested. 
> *The following query fails to validate:*
> {code:sql}
> select avg(sum(i_item_sk)) from item;
> {code}
> Error:
> Query failed: SqlValidatorException: Aggregate expressions cannot be nested
> Log attached. 
> Reference: TPCDS queries (20, 63, 98, ...) fail to execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413651#comment-15413651
 ] 

ASF GitHub Bot commented on DRILL-4795:
---

Github user amansinha100 commented on the issue:

https://github.com/apache/drill/pull/563
  
+1


> Nested aggregate windowed query fails - IllegalStateException 
> --
>
> Key: DRILL-4795
> URL: https://issues.apache.org/jira/browse/DRILL-4795
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
>Priority: Critical
> Attachments: tblWnulls.parquet
>
>
> The below two window function queries fail on MapR Drill 1.8.0 commit ID 
> 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> From drillbit.log
> {noformat}
> 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by 
> c2) from `tblWnulls.parquet`
> ...
> 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: java.lang.IllegalStateException: This generator does not support 
> mappings beyond
> at 
> org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) 
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
>  ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
>

[jira] [Commented] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413652#comment-15413652
 ] 

ASF GitHub Bot commented on DRILL-4795:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/563


> Nested aggregate windowed query fails - IllegalStateException 
> --
>
> Key: DRILL-4795
> URL: https://issues.apache.org/jira/browse/DRILL-4795
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node CentOS cluster
>Reporter: Khurram Faraaz
>Assignee: Gautam Kumar Parai
>Priority: Critical
> Attachments: tblWnulls.parquet
>
>
> The below two window function queries fail on MapR Drill 1.8.0 commit ID 
> 34ca63ba
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from 
> `tblWnulls.parquet`;
> Error: SYSTEM ERROR: IllegalStateException: This generator does not support 
> mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] 
> (state=,code=0)
> {noformat}
> From drillbit.log
> {noformat}
> 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO  
> o.a.drill.exec.work.foreman.Foreman - Query text for query id 
> 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by 
> c2) from `tblWnulls.parquet`
> ...
> 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: This generator does not support mappings beyond
> Fragment 0:0
> [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>  ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287)
>  [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_101]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: java.lang.IllegalStateException: This generator does not support 
> mappings beyond
> at 
> org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) 
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
>  ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
> org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at 
>

[jira] [Created] (DRILL-4838) Storage plugin directory location visibility in installation guide

2016-08-09 Thread Arnaud Linz (JIRA) 
Arnaud Linz created DRILL-4838:
--

 Summary: Storage plugin directory location visibility in 
installation guide
 Key: DRILL-4838
 URL: https://issues.apache.org/jira/browse/DRILL-4838
 Project: Apache Drill
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.7.0
 Environment: Windows 7 64 bits
Reporter: Arnaud Linz
Priority: Minor


For security reasons, my system prevents any user (but the administrator) to 
write at the C:\ level. 
When following the installation instruction and starting Drill embedded for 
windows, it tries to create a storage in /tmp (it does not use the environment 
variables such as TEMP or TMP) and fails.

There is a variable sys.store.provider.local.path that can be set to solve that 
problem, but it lacks visibility in the installation guide (it is documented 
further in the configuration section).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4837) go odbc 连接drill，默认会在sql后面加上limit 0

2016-08-09 Thread xwinie (JIRA) 
xwinie created DRILL-4837:
-

 Summary: go odbc 连接drill，默认会在sql后面加上limit 0
 Key: DRILL-4837
 URL: https://issues.apache.org/jira/browse/DRILL-4837
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - ODBC
Affects Versions: 1.6.0, 1.7.0
 Environment: win10odbc64bitgo 1.6  
Reporter: xwinie
 Fix For: Future


hi：
我用go odbc 连接drill，默认会在sql后面加上limit 0，go语言的odbc连接其他数据库，并无此情况。
go ：
connStr := `Driver={MapR Drill ODBC Driver};
Catalog=DRILL; AuthenticationType=No Authentication; 
ConnectionType=Direct;
Host=192.168.10.241;
Port=31010;`

db, err := sql.Open("odbc", connStr)
if err != nil {
panic(err)
}
defer db.Close()

sql := "select * from  mysqlplugin.mfm.sys_dict"

fmt.Println(sql)


rows, err := db.Query(sql)
if err != nil {
panic(err)
}


drill：

2016-08-09 14:12:18,989 [28568bbd-0748-9eca-8a7e-a595d7fc4ed0:foreman] INFO  
o.a.drill.exec.work.foreman.Foreman - Query text for query id 
28568bbd-0748-9eca-8a7e-a595d7fc4ed0: SELECT * FROM (select * from  
mysqlplugin.mfm.sys_dict where 1=1) T LIMIT 0




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)