Re: [ANNOUNCE] New PMC member: Arina Ielchiieva

[Sudheesh Katkam]

Congratulations and thank you, Arina.

On Wed, Aug 2, 2017 at 1:38 PM, Paul Rogers  wrote:

> The success of the Drill 1.11 release proves this is a well-deserved move.
> Congratulations!
>
> - Paul
>
> > On Aug 2, 2017, at 11:23 AM, Aman Sinha  wrote:
> >
> > I am pleased to announce that Drill PMC invited Arina Ielchiieva to the
> PMC
> > and she has accepted the invitation.
> >
> > Congratulations Arina and thanks for your contributions !
> >
> > -Aman
> > (on behalf of Drill PMC)
>
>

Re: [ANNOUNCE] New Committer: Paul Rogers

[Sudheesh Katkam]

Forgot to mention, not many developers know about this: 
https://github.com/paul-rogers/drill/wiki

So thank you Paul, for that informative wiki, and all your contributions.

On May 19, 2017, at 10:50 AM, Paul Rogers 
> wrote:

Thanks everyone!

- Paul

On May 19, 2017, at 10:30 AM, Kunal Khatua 
> wrote:

Congratulations, Paul !!  Thank you for your contributions!


From: Khurram Faraaz >
Sent: Friday, May 19, 2017 10:07:09 AM
To: dev
Subject: Re: [ANNOUNCE] New Committer: Paul Rogers

Congratulations, Paul!


From: Bridget Bevens >
Sent: Friday, May 19, 2017 10:29:29 PM
To: dev
Subject: Re: [ANNOUNCE] New Committer: Paul Rogers

Congratulations, Paul!


From: Jinfeng Ni >
Sent: Friday, May 19, 2017 9:57:35 AM
To: dev
Subject: Re: [ANNOUNCE] New Committer: Paul Rogers

Congratulations, Paul!


On Fri, May 19, 2017 at 9:36 AM, Aman Bawa 
> wrote:

Congratulations, Paul!

On 5/19/17, 8:22 AM, "Aman Sinha" 
> wrote:

  The Project Management Committee (PMC) for Apache Drill has invited
Paul
  Rogers to become a committer, and we are pleased to announce that he
has
  accepted.

  Paul has a long list of contributions that have touched many aspects
of the
  product.

  Welcome Paul, and thank you for your contributions.  Keep up the good
work !

  - Aman

  (on behalf of the Apache Drill PMC)

Re: [ANNOUNCE] New Committer: Paul Rogers

[Sudheesh Katkam]

Congratulations, Paul!

> On May 19, 2017, at 9:46 AM, Robert Hou  wrote:
> 
> Congrats, Paul!
> 
> 
> 
> From: Chunhui Shi 
> Sent: Friday, May 19, 2017 9:44 AM
> To: dev
> Subject: Re: [ANNOUNCE] New Committer: Paul Rogers
> 
> Congrats Paul! Thank you for your contributions!
> 
> 
> From: rahul challapalli 
> Sent: Friday, May 19, 2017 9:20:52 AM
> To: dev
> Subject: Re: [ANNOUNCE] New Committer: Paul Rogers
> 
> Congratulations Paul. Well Deserved.
> 
> On Fri, May 19, 2017 at 8:46 AM, Gautam Parai  wrote:
> 
>> Congratulations Paul and thank you for your contributions!
>> 
>> 
>> Gautam
>> 
>> 
>> From: Abhishek Girish 
>> Sent: Friday, May 19, 2017 8:27:05 AM
>> To: dev@drill.apache.org
>> Subject: Re: [ANNOUNCE] New Committer: Paul Rogers
>> 
>> Congrats Paul!
>> 
>> On Fri, May 19, 2017 at 8:23 AM, Charles Givre  wrote:
>> 
>>> Congrats Paul!!
>>> 
>>> On Fri, May 19, 2017 at 11:22 AM, Aman Sinha 
>> wrote:
>>> 
 The Project Management Committee (PMC) for Apache Drill has invited
>> Paul
 Rogers to become a committer, and we are pleased to announce that he
>> has
 accepted.
 
 Paul has a long list of contributions that have touched many aspects of
>>> the
 product.
 
 Welcome Paul, and thank you for your contributions.  Keep up the good
>>> work
 !
 
 - Aman
 
 (on behalf of the Apache Drill PMC)
 
>>> 
>> 

Upgrading Netty

[Sudheesh Katkam]

Hi all,

As part of working on DRILL-5431 [1], I found a bug in Netty [2], which is due 
to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does anyone 
foresee issues with upgrading to the latest version of Netty? I noticed Apache 
Arrow upgraded to 4.0.41 [5].

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-5431
[2] https://github.com/netty/netty/issues/6709
[3] https://github.com/netty/netty/pull/6713
[4] https://github.com/apache/drill/blob/master/pom.xml#L550
[5] 
https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3525c192745313b3da48

Re: DRAFT - ASF Board report for Apache Drill (May 2017)

[Sudheesh Katkam]

+1

> On May 1, 2017, at 2:10 AM, Parth Chandra <par...@apache.org> wrote:
> 
> Hi Drill devs,
> 
> The quarterly report to the Apache Board is due again. Below is a draft of
> the report I plan to file.
> Any suggestions/additions from the dev team? I'll file this by the 3rd so
> please provide comments by then.
> 
> Thanks
> 
> Parth
> 
> --- Report Begins ---
> 
> ## Description:
> - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> Storage
> 
> ## Issues:
> - There are no issues requiring board attention at this time
> 
> ## Activity:
> - Since the last board report, Drill has released version 1.10
> - Drill has added many improvements since the last report including
>- Support for left outer joins with nested loop joins
>– Support for ANSI_QUOTES option to allow alternatives to backtick for
> quoting strings
>– New sub-operator test framework
>– Fixed missing query text in prepared statement
>– Fixed new external sort when data contains map type columns
>– Improved query planning time against MapR-DB tables via caching of
> row count metadata
>– Improved query planning time by using runtime metadata dispatchers
> 
> ## Health report:
> - The project is healthy. Development activity is at the same level as the
> previous period. Activity on the dev mailing list, JIRAs, and pull requests
> is the same or higher than in the previous period. Three new committers
> were added in the last period.
> 
> 
> ## PMC changes:
> 
> - Currently 17 PMC members.
> - No new PMC members added in the last 3 months
> - One PMC member has resigned due to lack of time.
> - Last PMC addition was Sudheesh Katkam on Wed Oct 05 2016
> 
> ## Committer base changes:
> 
> - Currently 33 committers.
> - New commmitters:
>- Abhishek Girish was added as a committer on Mon Feb 13 2017
>- Arina Ielchiieva was added as a committer on Thu Feb 23 2017
>- Rahul Kumar Challapalli was added as a committer on Wed Feb 15 2017
> 
> ## Releases:
> 
> - 1.10.0 was released on Tue Mar 14 2017
> 
> ## Mailing list activity:
> 
> 
> - dev@drill.apache.org:
>- 436 subscribers (up 0 in the last 3 months):
>- 2066 emails sent to list (1758 in previous quarter)
> 
> - iss...@drill.apache.org:
>- 20 subscribers (up 0 in the last 3 months):
>- 3118 emails sent to list (2499 in previous quarter)
> 
> - u...@drill.apache.org:
>- 586 subscribers (up 9 in the last 3 months):
>- 382 emails sent to list (374 in previous quarter)
> 
> 
> ## JIRA activity:
> 
> - 220 JIRA tickets created in the last 3 months
> - 166 JIRA tickets closed/resolved in the last 3 months
> 
> --- Report Ends ---

Re: Fwd: Re: Apache Drill Repository is Empty!

[Sudheesh Katkam]

Back to normal now.


From: Sudheesh Katkam <sudhe...@apache.org>
Sent: Friday, April 28, 2017 9:45:33 AM
To: dev@drill.apache.org
Subject: Fwd: Re: Apache Drill Repository is Empty!

FYI. I had reached out to GitHub about github.com/apache/drill being empty.

-- Forwarded message --
From: "Laura Coursen (GitHub Staff)" <supp...@github.com>
Date: Apr 28, 2017 9:35 AM
Subject: Re: Apache Drill Repository is Empty!
To: "Sudheesh Katkam" <sudhe...@apache.org>
Cc:

Hi there,

Sorry for the trouble! We've now had a couple of reports of this problem,
and we've opened an issue internally to investigate.

I don't have an ETA on a fix, but we'll be in touch if we need more
information from you or if we have any information to share.

Regards,
Laura
@lecoursen
GitHub Support

"hubot" user deleted all the branches for this Apache repository:

https://github.com/apache/drill

Looks like a lot of other Apache projects have the same issue:

https://github.com/apache

Fwd: Re: Apache Drill Repository is Empty!

[Sudheesh Katkam]

FYI. I had reached out to GitHub about github.com/apache/drill being empty.

-- Forwarded message --
From: "Laura Coursen (GitHub Staff)" <supp...@github.com>
Date: Apr 28, 2017 9:35 AM
Subject: Re: Apache Drill Repository is Empty!
To: "Sudheesh Katkam" <sudhe...@apache.org>
Cc:

Hi there,

Sorry for the trouble! We've now had a couple of reports of this problem,
and we've opened an issue internally to investigate.

I don't have an ETA on a fix, but we'll be in touch if we need more
information from you or if we have any information to share.

Regards,
Laura
@lecoursen
GitHub Support

"hubot" user deleted all the branches for this Apache repository:

https://github.com/apache/drill

Looks like a lot of other Apache projects have the same issue:

https://github.com/apache

[jira] [Resolved] (DRILL-5322) Provide an OperatorFixture for sub-operator unit testing setup

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-5322.

Resolution: Fixed

Fixed as part of DRILL-5318.

> Provide an OperatorFixture for sub-operator unit testing setup
> --
>
> Key: DRILL-5322
> URL: https://issues.apache.org/jira/browse/DRILL-5322
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> We recently created various "fixture" classes to assist with system-level 
> testing: {{LogFixture}}, {{ClusterFixture}} and {{ClientFixture}}. Each 
> handles the tedious work of setting up the conditions to run certain kinds of 
> tests.
> In the same way, we need an {{OperatorFixture}} to set up the low-level bits 
> and pieces needed for operator-level, and sub-operator-level unit testing.
> The {{DrillConfig}} is used by both the system-level and operator-level 
> fixtures. So, pull the config-setup tasks our of (cluster) {{FixtureBuilder}} 
> (should rename) and into a new {{ConfigBuilder}}. Leave the existing methods 
> in {{FixtureBuilder}}, but modify them to be wrappers around the new config 
> builder.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (DRILL-5321) Refactor FragmentContext for unit testing

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-5321.

Resolution: Fixed

See DRILL-5319.

> Refactor FragmentContext for unit testing
> -
>
> Key: DRILL-5321
> URL: https://issues.apache.org/jira/browse/DRILL-5321
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Each operator has visibility to the {{FragmentContext}} class. 
> {{FragmentContext}} provides access to all of Drill internals: the Drillbit 
> context, the network interfaces, RPC messages and so on.
> Further, all the code generation mechanisms require a {{FragmentContext}} 
> object.
> This structure creates a large barrier to unit testing. To test, say, a 
> particular bit of generated code, we must have the entire Drillbit running so 
> we can obtain a {{FragmentContext}}. Clearly, this is less than ideal.
> Upon inspection, it turns out that the {{FragmentContext}} is mostly needed, 
> by many operators, to generate code. Of the many methods in 
> {{FragmentContext}}, code generation uses only six.
> The solution is to create a new super-interface, {{CodeGenContext}}, which 
> holds those six methods. The {{CodeGenContext}} can be easily re-implemented 
> for unit testing.
> Then, modify all the code-generation classes that currently take 
> {{FragmentContext}} to take {{CodeGenContext}} instead.
> Since {{FragmentContext}} derives from {{CodeGenContext}}, existing operator 
> code "just works."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (DRILL-5433) Authentication failed: Server requires authentication using [kerberos, plain]

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-5433.

   Resolution: Not A Problem
Fix Version/s: (was: 1.10.0)

Closing as "not a problem" because the issues are system configuration related.

> Authentication failed: Server requires authentication using [kerberos, plain]
> -
>
> Key: DRILL-5433
> URL: https://issues.apache.org/jira/browse/DRILL-5433
> Project: Apache Drill
>  Issue Type: Task
>  Components: Functions - Drill
>Affects Versions: 1.10.0
> Environment: OS: Redhat Linux 6.7, HDP 2.5.3, Kerberos enabled, 
> Hardware: VmWare
>Reporter: Parag Darji
>Priority: Minor
>  Labels: newbie, security
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I've setup Apace drill 1.10.0 on RHEL 6.7, HDP 2.5.3, kerberos enabled
> I'm getting below error while running "drill-conf" or sqlline as user "drill" 
> which is configured in the "drill-override.conf" file. 
> {code}
> drill@host:/opt/drill/bin>  drill-conf
> Error: Failure in connecting to Drill: 
> org.apache.drill.exec.rpc.NonTransientRpcException: 
> javax.security.sasl.SaslException: Authentication failed: Server requires 
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by 
> javax.security.sasl.SaslException: Server requires authentication using 
> [kerberos, plain]. Insufficient credentials?] (state=,code=0)
> java.sql.SQLException: Failure in connecting to Drill: 
> org.apache.drill.exec.rpc.NonTransientRpcException: 
> javax.security.sasl.SaslException: Authentication failed: Server requires 
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by 
> javax.security.sasl.SaslException: Server requires authentication using 
> [kerberos, plain]. Insufficient credentials?]
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:166)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at 
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at 
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: org.apache.drill.exec.rpc.NonTransientRpcException: 
> javax.security.sasl.SaslException: Authentication failed: Server requires 
> authentication using [kerberos, plain]. Insufficient credentials? [Caused by 
> javax.security.sasl.SaslException: Server requires authentication using 
> [kerberos, plain]. Insufficient credentials?]
> at 
> org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:157)
> at 
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at 
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:157)
> ... 18 more
> Caused by: javax.security.sasl.SaslException: Authentication failed: Server 
> requires authentication using [kerberos, plain]. Insufficient credentials? 
> [Caused by javax.security.sasl.SaslException: Server requires authentication 
> using [kerberos, plain]. Insufficient credentials?]
> at 
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:204)
> at 
> org.apache.drill.exec.rpc.user.UserClient$3.mapException(UserClient.java:197)
> 

[jira] [Created] (DRILL-5439) For Kerberos, Support "_host" replacement in "principal" Connection Parameter

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5439:
--

 Summary: For Kerberos, Support "_host" replacement in "principal" 
Connection Parameter
 Key: DRILL-5439
 URL: https://issues.apache.org/jira/browse/DRILL-5439
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - C++, Client - Java
    Reporter: Sudheesh Katkam
Priority: Minor


(requirement from [this 
comment|https://issues.apache.org/jira/browse/DRILL-5433?focusedCommentId=15971877])

"_host" can be specified in drill-override.conf; this replaces "_host" with 
FQDN of that host.

If "_host" is specified as service name in connection parameter (using 
"principal" or "service_host"), use the value for "drillbit" property or 
get/use a drillbit hostname from zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[HANGOUT] Topics for 04/18/17

[Sudheesh Katkam]

Hi drillers,

Our bi-weekly hangout is tomorrow (04/18/17, 10 AM PT). If you have any 
suggestions for hangout topics, you can add them to this thread. We will also 
ask around at the beginning of the hangout for topics.

Hangout link: 
https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

Thank you,
Sudheesh

[jira] [Created] (DRILL-5431) Support SSL

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5431:
--

 Summary: Support SSL
 Key: DRILL-5431
 URL: https://issues.apache.org/jira/browse/DRILL-5431
 Project: Apache Drill
  Issue Type: New Feature
  Components: Client - Java, Client - ODBC
Reporter: Sudheesh Katkam


Support SSL between Drillbit and JDBC/ODBC drivers. Drill already supports 
HTTPS for web traffic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5425) Support HTTP Kerberos auth using SPNEGO

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5425:
--

 Summary: Support HTTP Kerberos auth using SPNEGO
 Key: DRILL-5425
 URL: https://issues.apache.org/jira/browse/DRILL-5425
 Project: Apache Drill
  Issue Type: New Feature
  Components: Web Server
Reporter: Sudheesh Katkam


DRILL-4280 supports Kerberos through JDBC and ODBC API. This ticket requests to 
add Kerberos (using [SPENGO|https://en.wikipedia.org/wiki/SPNEGO]) for HTTP 
connections.

This requires creating "direct" web sessions; currently web sessions are 
sessions over Java client sessions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5387) TestBitBitKerberos and TestUserBitKerberos cause sporadic unit test failures

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5387:
--

 Summary: TestBitBitKerberos and TestUserBitKerberos cause sporadic 
unit test failures
 Key: DRILL-5387
 URL: https://issues.apache.org/jira/browse/DRILL-5387
 Project: Apache Drill
  Issue Type: Bug
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam


TestOptionsAuthEnabled and TestInboundImpersonation sporadically fail. There is 
a [Java 
trick|https://github.com/apache/drill/blob/master/exec/java-exec/src/test/java/org/apache/hadoop/security/UgiTestUtil.java#L29]
 to reset some static state in TestUserBitKerberos and TestBitBitKerberos to 
ensure the JVM is reusable for other tests as done in the [Hadoop auth 
tests|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGIWithMiniKdc.java#L53],
 but this trick does not seem to work always. So disable these tests. In the 
future, maybe the tests can be run separately through surefire but not as part 
of the default build?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (DRILL-5088) Error when reading DBRef column

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-5088.

   Resolution: Fixed
Fix Version/s: 1.10.0

Fixed in 
[b892b99|https://github.com/apache/drill/commit/b892b997dfa0259550942f076b0afd89b27c9fdf]

> Error when reading DBRef column
> ---
>
> Key: DRILL-5088
> URL: https://issues.apache.org/jira/browse/DRILL-5088
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Data Types
> Environment: drill 1.9.0
> mongo 3.2
>Reporter: Guillaume Champion
>Assignee: Chunhui Shi
> Fix For: 1.10.0
>
>
> In a mongo database with DBRef, when a DBRef is inserted in the first line of 
> a mongo's collection drill query failed :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> {code}
> Simple example to reproduce:
> In mongo instance
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> Error: SYSTEM ERROR: CodecConfigurationException: Can't find a codec for 
> class com.mongodb.DBRef.
> [Error Id: 2944d766-e483-4453-a706-3d481397b186 on Analytics-Biznet:31010] 
> (state=,code=0)
> {code}
> If the first line doesn't contain de DBRef, drill will querying correctly :
> In a mongo instance :
> {code}
> db.contact2.drop();
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8939") });
> db.contact2.insert({ "_id" : ObjectId("582081d96b69060001fd8938"), "account" 
> : DBRef("contact", ObjectId("999cbf116b69060001fd8611")) });
> {code}
> In drill :
> {code}
> 0: jdbc:drill:zk=local> select * from mongo.mydb.contact2;
> +--+---+
> | _id  |account   
>  |
> +--+---+
> | {"$oid":"582081d96b69060001fd8939"}  | {"$id":{}}   
>  |
> | {"$oid":"582081d96b69060001fd8938"}  | 
> {"$ref":"contact","$id":{"$oid":"999cbf116b69060001fd8611"}}  |
> +--+---+
> 2 rows selected (0,563 seconds)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[ANNOUNCE] New Committer: Arina Ielchiieva

[Sudheesh Katkam]

The Project Management Committee (PMC) for Apache Drill has invited Arina
Ielchiieva to become a committer, and we are pleased to announce that she
has accepted.

Arina has a long list of contributions [1] that have touched many aspects
of the product. Her work includes features such as dynamic UDF support and
temporary tables support.

Welcome Arina, and thank you for your contributions.

- Sudheesh, on behalf of the Apache Drill PMC

[1] https://github.com/apache/drill/commits/master?author=arina-ielchiieva

Re: Time for 1.10 release

[Sudheesh Katkam]

I would like to include:

+ DRILL-4280: https://github.com/apache/drill/pull/578 (going through the last 
round of comments)

Thank you,
Sudheesh

> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  wrote:
> 
> Hi Drillers,
> 
> It has been almost 3 months since we release Drill 1.9. We have
> resolved plenty of fixes and improvements (closed around 88 JIRAs
> [1]). I propose that we start the 1.10 release process, and set
> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we should
> start build a release candidate.
> 
> Please reply in this email thread if you have something near complete
> and you would like to include in 1.10 release.
> 
> I volunteer as the release manager, unless someone else come forward.
> 
> Thanks,
> 
> Jinfeng
> 
> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/12338769

Re: [ANNOUNCE] New Apache Drill Committer - Abhishek Girish

[Sudheesh Katkam]

Congratulations, Abhishek.

> On Feb 15, 2017, at 1:22 PM, Aditya  wrote:
> 
> Congratulations, Abhishek!
> 
> On Tue, Feb 14, 2017 at 10:39 PM, Khurram Faraaz  wrote:
> 
>> Congrats Abhishek!
>> 
>> 
>> From: Parth Chandra 
>> Sent: Wednesday, February 15, 2017 9:48:18 AM
>> To: dev@drill.apache.org
>> Subject: [ANNOUNCE] New Apache Drill Committer - Abhishek Girish
>> 
>> The Project Management Committee (PMC) for Apache Drill has invited
>> Abhishek Girish to become a committer and we are pleased  to announce that
>> he has accepted.
>> 
>> Welcome Abhishek and thanks for your great contributions!
>> 
>> Parth
>> 

Re: [ANNOUNCE] NewApache Drill Committer - Rahul Challapalli

[Sudheesh Katkam]

Congratulations, Rahul!

> On Feb 15, 2017, at 1:22 PM, Aditya  wrote:
> 
> Congratulations, Rahul!
> 
> On Wed, Feb 15, 2017 at 1:04 PM, rahul challapalli <
> challapallira...@gmail.com> wrote:
> 
>> Thank you Ramana and Khurram
>> 
>> On Feb 15, 2017 11:19 AM, "Khurram Faraaz"  wrote:
>> 
>>> Congrats Rahul!
>>> 
>>> 
>>> From: Parth Chandra 
>>> Sent: Wednesday, February 15, 2017 9:48:15 AM
>>> To: dev@drill.apache.org
>>> Subject: [ANNOUNCE] NewApache Drill Committer - Rahul Challapalli
>>> 
>>> The Project Management Committee (PMC) for Apache Drill has invited Rahul
>>> Challapalli to become a committer and we are pleased  to announce that he
>>> has accepted.
>>> 
>>> Welcome Rahul and thanks for your great contributions!
>>> 
>>> Parth
>>> 
>> 

Re: [DRAFT REPORT] Apache Drill

[Sudheesh Katkam]

LGTM

One more feature: ability to add UDFs dynamically

> On Feb 3, 2017, at 10:47 AM, Parth Chandra <par...@apache.org> wrote:
> 
> It's time to file the Apache Drill quarterly report with the Board. Below
> is the draft of the report.
> 
> Is there anything folks would like to add?
> 
> Thanks
> 
> Parth
> 
> --- Report Begins 
> ## Description:
> - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> Storage
> 
> ## Issues:
> - There are no issues requiring board attention at this time
> 
> ## Activity:
> - Since the last board report, Drill has released version 1.9
> - Drill has added many new features since the last report. More Parquet
> reader performance improvements, temp tables support, an improved work
> assignment algorithm,  and an httpd format plugin.
> - Work continues on improved use of statistics, and security enhancements
> (including support for Kerberos) and a sort with managed memory usage.
> 
> ## Health report:
> - The project is healthy. Development activity is high and is reflected in
> an increase in the number of mails to the mailing list, many new pull
> requests and increased activity in JIRA. Two new committers were added in
> the last period.
> 
> ## PMC changes:
> 
> - Currently 18 PMC members.
> - No new PMC members added in the last 3 months
> - Last PMC addition was Sudheesh Katkam on Wed Oct 05 2016
> 
> ## Committer base changes:
> 
> - Currently 30 committers.
> - New commmitters:
>- Chris Westin was added as a committer on Wed Nov 30 2016
>- Neeraja Rentachintala was added as a committer on Wed Nov 16 2016
> 
> ## Releases:
> 
> - 1.9.0 was released on Mon Nov 28 2016
> 
> ## Mailing list activity:
> 
> - Mailing list activity is healthy.
> 
> - dev@drill.apache.org:
>- 436 subscribers (up 2 in the last 3 months):
>- 1825 emails sent to list (1652 in previous quarter)
> 
> - iss...@drill.apache.org:
>- 20 subscribers (up 0 in the last 3 months):
>- 2524 emails sent to list (2068 in previous quarter)
> 
> - u...@drill.apache.org:
>- 578 subscribers (up 13 in the last 3 months):
>- 372 emails sent to list (461 in previous quarter)
> 
> 
> ## JIRA activity:
> 
> - 238 JIRA tickets created in the last 3 months
> - 87 JIRA tickets closed/resolved in the last 3 months
> 
> 
> --- Report Ends 

[jira] [Created] (DRILL-5218) Support Disabling Heartbeats in C++ Client

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5218:
--

 Summary: Support Disabling Heartbeats in C++ Client
 Key: DRILL-5218
 URL: https://issues.apache.org/jira/browse/DRILL-5218
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam


Heartbeats between bits allow for detecting health of remotes, but heartbeats 
between client and bit are not necessary. So allow to (at least) disable 
heartbeats between C++ client and bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5217) Heartbeat Fails when C++ client receives a large ResultSet

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5217:
--

 Summary: Heartbeat Fails when C++ client receives a large ResultSet
 Key: DRILL-5217
 URL: https://issues.apache.org/jira/browse/DRILL-5217
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Sudheesh Katkam
Priority: Critical


If the listener thread is occupied for longer than 15 seconds (heartbeat 
timeout) while [handling a message from the 
drillbit|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L1286]
 e.g. [processing query data blocks if the query result listener's buffer is 
full|https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L899],
 heartbeats fail because the same thread is responsible for sending heartbeats!

Fix is to [handle long running 
operations|http://stackoverflow.com/questions/17648725/long-running-blocking-operations-in-boost-asio-handlers]
 separately using boost asio.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [HANGOUT] Topics for 01/24/17

[Sudheesh Katkam]

Join us here: 
https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

On Jan 23, 2017, at 6:43 PM, Sudheesh Katkam 
<skat...@mapr.com<mailto:skat...@mapr.com>> wrote:

I meant 01/24/17, 10 AM PT.

On Jan 23, 2017, at 12:43 PM, Sudheesh Katkam 
<skat...@mapr.com<mailto:skat...@mapr.com>> wrote:

Hi drillers,

Our bi-weekly hangout is tomorrow (01/23/17, 10 AM PT). If you have any 
suggestions for hangout topics, you can add them to this thread. We will also 
ask around at the beginning of the hangout for topics.

Thank you,
Sudheesh

Re: [HANGOUT] Topics for 01/24/17

[Sudheesh Katkam]

I meant 01/24/17, 10 AM PT.

> On Jan 23, 2017, at 12:43 PM, Sudheesh Katkam <skat...@mapr.com> wrote:
> 
> Hi drillers,
> 
> Our bi-weekly hangout is tomorrow (01/23/17, 10 AM PT). If you have any 
> suggestions for hangout topics, you can add them to this thread. We will also 
> ask around at the beginning of the hangout for topics.
> 
> Thank you,
> Sudheesh

[HANGOUT] Topics for 01/23/17

[Sudheesh Katkam]

Hi drillers,

Our bi-weekly hangout is tomorrow (01/23/17, 10 AM PT). If you have any 
suggestions for hangout topics, you can add them to this thread. We will also 
ask around at the beginning of the hangout for topics.

Thank you,
Sudheesh

Re: Invoking UDF that doesn't have parameters without paranthesis

[Sudheesh Katkam]

I do not know the exact answer.

Quite likely the Calcite parser needs to be changed, this blob:

https://github.com/apache/calcite/blob/master/core/src/main/codegen/templates/Parser.jj#L4765
 


Drill extends the Calcite parser, not sure if the change can be made here:

https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/includes/parserImpls.ftl
 


Thank you,
Sudheesh

> On Dec 19, 2016, at 3:00 PM, Nagarajan Chinnasamy 
>  wrote:
> 
> Hi Julian Hyde,
> 
> Thanks for your response. I am looking for the Drill way of doing it.
> 
> In fact I have coded session_id function exactly as one of other
> ContextFunctions of Drill. I seem to be missing something :(
> 
> 
> 
> 
> Best Regards,
> Nagu.
> 
> On Mon, Dec 19, 2016 at 1:30 PM, Nagarajan Chinnasamy <
> nagarajanchinnas...@gmail.com> wrote:
> 
>> Hi,
>> 
>> Am developing a UDF called SESSION_ID as detailed in issue:
>> 
>>https://issues.apache.org/jira/browse/DRILL-5043
>> 
>> It does not take any input parameters. Now, I can only invoke it using
>> parenthesis as in:
>> 
>>SELECT SESSION_ID() FROM (Values(1));
>> 
>> I would like to know what do I need to do if I need to invoke it without
>> paranthesis like:
>> 
>>SELECT SESSION_ID FROM (Values(1));
>> 
>> Appreciate your input.
>> 
>> Best Regards,
>> Nagu.
>> 

[jira] [Created] (DRILL-5129) Use SilentListener in DrillSeparatePlanningTest

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5129:
--

 Summary: Use SilentListener in DrillSeparatePlanningTest
 Key: DRILL-5129
 URL: https://issues.apache.org/jira/browse/DRILL-5129
 Project: Apache Drill
  Issue Type: Improvement
  Components: Tools, Build & Test
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Priority: Minor


[DrillSeparatePlanningTest.java|https://github.com/apache/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/DrillSeparatePlanningTest.java#L184]
 should use SilentListener in all tests, instead of PrintingResultsListener.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4812) Wildcard queries fail on Windows

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4812.

   Resolution: Fixed
Fix Version/s: 1.10.0

Fixed in 
[b656128|https://github.com/apache/drill/commit/b6561289998da6023efa864212033fbf5ac84cbd]

> Wildcard queries fail on Windows
> 
>
> Key: DRILL-4812
> URL: https://issues.apache.org/jira/browse/DRILL-4812
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.7.0
> Environment: Windows 7
>Reporter: Mike Lavender
>  Labels: easyfix, easytest, ready-to-commit, windows
> Fix For: 1.10.0
>
>
> Wildcards within the path of a query are not handled on windows and result in 
> a "String index out of range" exception.
> for example:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT SUM(qty) as num FROM 
> dfs.parquet.`/trends/2016/1/*/*/3701`;
> Error: VALIDATION ERROR: String index out of range: -1
> SQL Query null
> {noformat}
> 
> The problem exists within:
> exec\java-exec\src\main\java\org\apache\drill\exec\store\dfs\FileSelection.java
> private static Path handleWildCard(final String root)
> This function is looking for the index of the system specific PATH_SEPARATOR 
> which on windows is '\' (from System.getProperty("file.separator")).  The 
> path passed in to handleWildcard will not ever have those type of path 
> separators as the Path constructor (from org.apache.hadoop.fs.Path) sets all 
> the path separators to '/'.
> NOTE:
> private static String removeLeadingSlash(String path)
> in that same file explicitly looks for '/' and does not use the system 
> specific PATH_SEPARATOR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4987) Use ImpersonationUtil in RemoteFunctionRegistry

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4987.

   Resolution: Fixed
Fix Version/s: 1.10.0

Fixed in 
[a33a185|https://github.com/apache/drill/commit/a33a1858949cbac3a9c2f636a02c0b7c5bc25906]

> Use ImpersonationUtil in RemoteFunctionRegistry
> ---
>
> Key: DRILL-4987
> URL: https://issues.apache.org/jira/browse/DRILL-4987
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Functions - Drill
>        Reporter: Sudheesh Katkam
>    Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.10.0
>
>
> + Use ImpersonationUtil#getProcessUserName rather than  
> UserGroupInformation#getCurrentUser#getUserName in RemoteFunctionRegistry
> + Expose process users' group info in ImpersonationUtil and use that in 
> RemoteFunctionRegistry, rather than 
> UserGroupInformation#getCurrentUser#getGroupNames



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] - New Apache Drill Committer - Chris Westin

[Sudheesh Katkam]

Congratulations, Chris!

> On Dec 1, 2016, at 10:30 AM, Aditya  wrote:
> 
> Congratulations Chris!
> 
> On Thu, Dec 1, 2016 at 9:56 AM, Parth Chandra  wrote:
> 
>> Congrats Chris. And thank you for all your cool contributions!
>> 
>> 
>> 
>> On Thu, Dec 1, 2016 at 8:54 AM, Jacques Nadeau  wrote:
>> 
>>> On behalf of the Apache Drill PMC, I am very pleased to announce that
>> Chris
>>> Westin has accepted the invitation to become a committer in the project.
>>> 
>>> Welcome Chris and thanks for your great contributions!
>>> 
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>> 

Re: [ANNOUNCE] Apache Drill 1.9.0 Released

[Sudheesh Katkam]

Thank you for the information, Sebastian.

This maybe our first time posting to the announce list. I posted on the
list to provide more visibility about the release, but I used the previous
announcement template.

I will add a note about the brief details to our release process
instructions.

- Sudheesh

On Wed, Nov 30, 2016 at 10:35 AM, sebb <seb...@gmail.com> wrote:

> What is the project about? Why should I be interested in it?
> [rhetorical questions]
>
> The Announce emails are sent to people not on the developer or user lists.
> Most will have no idea what the project is about.
>
> So the e-mails should contain at least brief details of what the
> product does, and some info on why the new release might be of
> interest to them.
>
> Readers should not have to click the link to find out the basic information
> (although of course it is useful to have such links for further detail).
>
> Please can you add that information to future announce mails?
>
> Thanks.
>
>
> On 29 November 2016 at 23:52, Sudheesh Katkam <sudhe...@apache.org> wrote:
> > On behalf of the Apache Drill community, I am happy to announce the
> release
> > of Apache Drill 1.9.0.
> >
> > For information about Apache Drill, and to get involved, visit the
> project
> > website:
> >
> > https://drill.apache.org/
> >
> > This release introduces new features and enhancements, including
> > asynchronous Parquet reader, Parquet filter pushdown, dynamic UDF support
> > and HTTPD format plugin. In all, 70 issues have been resolved.
> >
> > The binary and source artifacts are available here:
> >
> > https://drill.apache.org/download/
> >
> > Review the release notes for a complete list of fixes and enhancements:
> >
> > https://drill.apache.org/docs/apache-drill-1-9-0-release-notes/
> >
> > Thanks to everyone in the community who contributed to this release!
> >
> > - Sudheesh
>

[ANNOUNCE] Apache Drill 1.9.0 Released

[Sudheesh Katkam]

On behalf of the Apache Drill community, I am happy to announce the release
of Apache Drill 1.9.0.

For information about Apache Drill, and to get involved, visit the project
website:

https://drill.apache.org/

This release introduces new features and enhancements, including
asynchronous Parquet reader, Parquet filter pushdown, dynamic UDF support
and HTTPD format plugin. In all, 70 issues have been resolved.

The binary and source artifacts are available here:

https://drill.apache.org/download/

Review the release notes for a complete list of fixes and enhancements:

https://drill.apache.org/docs/apache-drill-1-9-0-release-notes/

Thanks to everyone in the community who contributed to this release!

- Sudheesh

[jira] [Created] (DRILL-5081) Excessive info level logging introduced in DRILL-4203

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-5081:
--

 Summary: Excessive info level logging introduced in DRILL-4203
 Key: DRILL-5081
 URL: https://issues.apache.org/jira/browse/DRILL-5081
 Project: Apache Drill
  Issue Type: Bug
Reporter: Sudheesh Katkam
Assignee: Vitalii Diravka


Excessive info level logging introduced in 
[8461d10|https://github.com/apache/drill/commit/8461d10b4fd6ce56361d1d826bb3a38b6dc8473c].
 A line is printed for every row group being read, and for every metadata file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] Apache Drill Version after 1.9.0, etc.

[Sudheesh Katkam]

Hi all,

-

(A) I had asked the question about what the release version should be after 
1.9.0. Since this is part of the next release plan, a vote is required based on 
the discussion. For approval, the vote requires a lazy majority of active 
committers over 3 days.

Here are some comments from that thread:

Quoting Paul:

> For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good idea.
> 
> At first it may seem odd to go to 1.10 from 1.9. Might people get confused 
> between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest 7-series 
> release is 7.0.72. Java is on 8u112. And so on.
> 
> I like the idea of moving to 2.0 later when the team introduces a major 
> change, rather than by default just because the numbers roll around. For 
> example, Hadoop when to 2.x when YARN was introduced. Impala appears to have 
> moved to 2.0 when they added Spill to disk for some (all?) operators.


Quoting Parth:

> Specifically what did you want to discuss about the release number after 1.9? 
>  Ordinarily you would just go to 2.0. The only reason for holding off on 2.0, 
> that I can think of, is if you want to make breaking changes in the 2.0 
> release and those are not going to be ready for the next release cycle. Are 
> any dev's planning on such breaking changes? If so we should discuss that (or 
> any other reason we might have for deferring 2.0) in a separate thread?
> I'm +0 on any version number we chose.


I am +1 on Paul’s suggestion for 1.10.0, unless, as Parth noted, we plan to 
make breaking changes in the next release cycle.

@Jacques, any comments? You had mentioned about this a while back [1].

-

(B) Until discussion on (A) is complete, which may take a while, I propose we 
move the master to 1.10.0-SNAPSHOT to unblock committing to master branch. If 
there are no objections, I will do this tomorrow, once 1.9.0 release artifacts 
are propagated.

-

(C) I noticed there are some changes committed to master branch before the 
commit that moves to the next snapshot version. Did we face this issue in the 
past? If so, how did we resolve the issue? Is 'force push' an option?

-

Thank you,
Sudheesh

[1] 
http://mail-archives.apache.org/mod_mbox/drill-dev/201604.mbox/%3CCAJrw0OTiXLnmW25K0aQtsVmh3A4vxfwZzvHntxeYJjPdd-PnYQ%40mail.gmail.com%3E
 

[RESULT][VOTE] Release Apache Drill 1.9.0 RC2

[Sudheesh Katkam]

The proposal passes.

Final tally:

3 binding +1s
+ Sudheesh
+ Parth
+ Aman

6 non-binding +1s
+ Khurram
+ Dechang
+ Gautam
+ Karthikeyan
+ Robert
+ Kunal

No 0s or -1s

I'll push the release artifacts, and send an announcement soon after the
artifacts propagate. Thanks to everyone involved!

Thank you,
Sudheesh

-- Forwarded message --
From: Kunal Khatua <kkha...@maprtech.com>
Date: Sat, Nov 26, 2016 at 12:12 PM
Subject: Re: [VOTE] Release Apache Drill 1.9.0 RC2
To: dev@drill.apache.org


+1 (non-binding)

Downloaded and built from source.
Ran a few manual tests related to the latest features in the release, and
all looked fine.

~ Kunal
On Tue 22-Nov-2016 3:49:35 PM, Robert Hou <r...@maprtech.com> wrote:
+1 (non-binding)

Downloaded and built from source
Ran tests for parquet filter pushdown
Ran some ODBC tests.

On Tue, Nov 22, 2016 at 1:26 PM, Karthikeyan Manivannan
kmanivan...@maprtech.com> wrote:

> +1 (non-binding)
>
> - Built source on my vm and ran the unit tests
> - Ran some simple queries on embedded Drill.
>
> Karthik
>
> On Tue, Nov 22, 2016 at 8:41 AM, Aman Sinha wrote:
>
> > +1 (binding)
> >
> > - Downloaded source on my Linux VM, did a build and ran unit tests
> > successfully.
> > - Downloaded and installed the binaries on my mac
> > - Ran a few sanity test queries against a TPC-DS Scale factor 1 data
set.
> > - Checked Web UI for query profiles.
> > - Checked Maven artifacts on repositories.apache.org
> >
> > On Mon, Nov 21, 2016 at 10:22 AM, Gautam Parai
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > built from source.
> > > ran unit tests.
> > > ran fixed-bugs (Drill-4862, 4906, 4763, 4986 , 4927) queries.
> > > ran some random queries.
> > >
> > > Sudheesh,
> > > Should we remove Drill-4941, Drill-4525 from the Release Notes as they
> > are
> > > not fixed?
> > >
> > > Gautam
> > >
> > > On Mon, Nov 21, 2016 at 9:19 AM, Parth Chandra
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > checked signatures src and rpms
> > > > built from src
> > > > ran unit tests
> > > > ran about a hundred Parquet queries
> > > > built client
> > > > ran a few queries
> > > > tested artifacts - built and ran a client app using 1.9.0 jdbc-all
> > > >
> > > >
> > > >
> > > > On Mon, Nov 21, 2016 at 8:02 AM, Dechang Gu
> wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - downloaded and built from the source
> > > > > - deployed on 10-node cluster
> > > > > - ran the TPC-H queries
> > > > > - checked the results and profiles
> > > > >
> > > > >
> > > > > LGTM.
> > > > >
> > > > > -Dechang
> > > > >
> > > > > On Fri, Nov 18, 2016 at 1:56 PM, Sudheesh Katkam
> > sudhe...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to propose the third release candidate (RC2) of
> Apache
> > > > > Drill,
> > > > > > version 1.9.0. Thanks to everyone who contributed to this
> release!
> > > > > >
> > > > > > + Compared to RC0, this release candidate does not contain
> > > DRILL-4373,
> > > > > due
> > > > > > to a regression (DRILL-5034).
> > > > > > + Compared to RC1, this release candidate contains DRILL-5047.
> > > > > >
> > > > > > + The release candidate covers a total of 72 resolved JIRAs [1].
> > This
> > > > is
> > > > > > fewer than what was mentioned for RC1, which was actually
> > incorrect.
> > > > > > + The tarball artifacts are hosted at [2], and the maven
> artifacts
> > > are
> > > > > > hosted at [3].
> > > > > > + This release candidate is based on commit
> > > > > > 4312d65bd5e0f68dc963ed722d0cdfd2628ea5f5 located at [4].
> > > > > > + The artifacts are signed with the key at [5].
> > > > > >
> > > > > > The vote ends at 2:00 PM PT, November 23rd, 2016. Please vote
> > early.
> > > > > >
> > > > > > Here's my vote: +1
> > > > > >
> > > > > > Thank you,
> > > > > > Sudheesh
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > > projectId=12313820=12337861
> > > > > > [2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc2/
> > > > > > [3] https://repository.apache.org/content/repositories/
> > > > > > orgapachedrill-1039/
> > > > > > [4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
> > > > > > [5] https://people.apache.org/keys/committer/sudheesh.asc
> > > > > >
> > > > >
> > > >
> > >
> >
>

[VOTE] Release Apache Drill 1.9.0 RC2

[Sudheesh Katkam]

Hi all,

I would like to propose the third release candidate (RC2) of Apache Drill,
version 1.9.0. Thanks to everyone who contributed to this release!

+ Compared to RC0, this release candidate does not contain DRILL-4373, due
to a regression (DRILL-5034).
+ Compared to RC1, this release candidate contains DRILL-5047.

+ The release candidate covers a total of 72 resolved JIRAs [1]. This is
fewer than what was mentioned for RC1, which was actually incorrect.
+ The tarball artifacts are hosted at [2], and the maven artifacts are
hosted at [3].
+ This release candidate is based on commit
4312d65bd5e0f68dc963ed722d0cdfd2628ea5f5 located at [4].
+ The artifacts are signed with the key at [5].

The vote ends at 2:00 PM PT, November 23rd, 2016. Please vote early.

Here's my vote: +1

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?
projectId=12313820=12337861
[2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc2/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1039/
[4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
[5] https://people.apache.org/keys/committer/sudheesh.asc

Re: [RESULT] [VOTE] Release Apache Drill 1.9.0 RC1

[Sudheesh Katkam]

I agree that the issue is not a regression. So I’ll go ahead with the getting 
the next release candidate out today.

- Sudheesh

> On Nov 18, 2016, at 9:39 AM, Jacques Nadeau <jacq...@dremio.com> wrote:
> 
> It sounds like the issue is constrained only to JDBC then, despite my
> previous concerns. It also isn't a regression. As such, I guess it it
> shouldn't really be a blocker to the release. When I first saw the trace, I
> thought it was related to the new parallelization changes and was a
> regression.
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Fri, Nov 18, 2016 at 9:28 AM, Sudheesh Katkam <skat...@maprtech.com 
> <mailto:skat...@maprtech.com>>
> wrote:
> 
>> Venki, could you please take a look, since you are most familiar with that
>> piece of code? Or anyone else wants to take a look?
>> 
>> The issue can be reproduced with a simple unit test. In
>> TestJdbcPluginWithDerbyIT, add this test. and then run “mvn install” in the
>> storage-jdbc sub-project.
>> 
>> @Test // DRILL-4984
>> public void limit0() throws Exception {
>>testNoResult("SELECT * FROM derby.DRILL_DERBY_TEST.PERSON LIMIT
>> 0");
>> }
>> 
>> In the ticket, Hogler suggested “adding a check for null in
>> FindHardDistributionScans.java @line 55 before calling getDrillTable()”.
>> But that check may not be sufficient (I could be wrong) because the check
>> does not imply if “contains” should be set to true/false. The call to
>> unwrap() returns a different type of table (not DrillTable or
>> DrillTranslatableTable), and that may need to be investigated.
>> 
>> Thank you,
>> Sudheesh
>> 
>>> On Nov 17, 2016, at 10:09 PM, Jacques Nadeau <jacq...@dremio.com> wrote:
>>> 
>>> It might make sense for someone to look at this jira before rolling
>> another
>>> release: DRILL-4984
>>> 
>>> The stacktrace looks like it might be an issue with the new hard
>>> parallelization algorithm which could potentially influence all sources.
>> It
>>> might not have shown up in traditional regression tests if those always
>>> have source/drillbit affinity (just a random guess).
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> On Thu, Nov 17, 2016 at 10:50 AM, Sudheesh Katkam <skat...@maprtech.com
>> <mailto:skat...@maprtech.com <mailto:skat...@maprtech.com>>>
>>> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I had not noticed that Gautam mentioned about a potential bug. That is a
>>>> -1 from me on the proposed candidate; the bug is a regression in
>> behavior.
>>>> I did not push the release artifacts until now, and the announcement is
>> not
>>>> out.
>>>> 
>>>> The issue is that the query profile is not displayed past the point of
>>>> failure (trying to show a changed string option). So I will propose
>> another
>>>> candidate once this issue is fixed [1, 2].
>>>> 
>>>> In the mean time, please test the candidate for other regressions.
>>>> 
>>>> Thank you,
>>>> Sudheesh
>>>> 
>>>> [1] https://issues.apache.org/jira/browse/DRILL-5047 <
>>>> https://issues.apache.org/jira/browse/DRILL-5047 <
>> https://issues.apache.org/jira/browse/DRILL-5047 
>> <https://issues.apache.org/jira/browse/DRILL-5047>>>
>>>> [2] https://github.com/apache/drill/pull/655 
>>>> <https://github.com/apache/drill/pull/655> <
>> https://github.com/apache/drill/pull/655 
>> <https://github.com/apache/drill/pull/655>> <https://github.com/apache/ 
>> <https://github.com/apache/> <
>> https://github.com/apache/ <https://github.com/apache/>>
>>>> drill/pull/655>
>>>> 
>>>>> On Nov 16, 2016, at 7:15 PM, Sudheesh Katkam <skat...@maprtech.com>
>>>> wrote:
>>>>> 
>>>>> The proposal passes!
>>>>> 
>>>>> Final tally:
>>>>> 
>>>>> 3 binding +1s
>>>>> + Sudheesh
>>>>> + Aman
>>>>> + Parth
>>>>> 
>>>>> 12 non-binding +1s
>>>>> + Khurram
>>>>> + Dechang
>>>>> + Rahul
>>>>> + Chunhui
>>>>> + Karthikeyan
>>>>> + Robert
>>>>> + Paul
>>>>> + Krystal
>>>>> + S

Re: [RESULT] [VOTE] Release Apache Drill 1.9.0 RC1

[Sudheesh Katkam]

Venki, could you please take a look, since you are most familiar with that 
piece of code? Or anyone else wants to take a look?

The issue can be reproduced with a simple unit test. In 
TestJdbcPluginWithDerbyIT, add this test. and then run “mvn install” in the 
storage-jdbc sub-project.

@Test // DRILL-4984
public void limit0() throws Exception {
testNoResult("SELECT * FROM derby.DRILL_DERBY_TEST.PERSON LIMIT 0");
}

In the ticket, Hogler suggested “adding a check for null in 
FindHardDistributionScans.java @line 55 before calling getDrillTable()”. But 
that check may not be sufficient (I could be wrong) because the check does not 
imply if “contains” should be set to true/false. The call to unwrap() returns a 
different type of table (not DrillTable or DrillTranslatableTable), and that 
may need to be investigated.

Thank you,
Sudheesh

> On Nov 17, 2016, at 10:09 PM, Jacques Nadeau <jacq...@dremio.com> wrote:
> 
> It might make sense for someone to look at this jira before rolling another
> release: DRILL-4984
> 
> The stacktrace looks like it might be an issue with the new hard
> parallelization algorithm which could potentially influence all sources. It
> might not have shown up in traditional regression tests if those always
> have source/drillbit affinity (just a random guess).
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Thu, Nov 17, 2016 at 10:50 AM, Sudheesh Katkam <skat...@maprtech.com 
> <mailto:skat...@maprtech.com>>
> wrote:
> 
>> Hi all,
>> 
>> I had not noticed that Gautam mentioned about a potential bug. That is a
>> -1 from me on the proposed candidate; the bug is a regression in behavior.
>> I did not push the release artifacts until now, and the announcement is not
>> out.
>> 
>> The issue is that the query profile is not displayed past the point of
>> failure (trying to show a changed string option). So I will propose another
>> candidate once this issue is fixed [1, 2].
>> 
>> In the mean time, please test the candidate for other regressions.
>> 
>> Thank you,
>> Sudheesh
>> 
>> [1] https://issues.apache.org/jira/browse/DRILL-5047 <
>> https://issues.apache.org/jira/browse/DRILL-5047 
>> <https://issues.apache.org/jira/browse/DRILL-5047>>
>> [2] https://github.com/apache/drill/pull/655 
>> <https://github.com/apache/drill/pull/655> <https://github.com/apache/ 
>> <https://github.com/apache/>
>> drill/pull/655>
>> 
>>> On Nov 16, 2016, at 7:15 PM, Sudheesh Katkam <skat...@maprtech.com>
>> wrote:
>>> 
>>> The proposal passes!
>>> 
>>> Final tally:
>>> 
>>> 3 binding +1s
>>> + Sudheesh
>>> + Aman
>>> + Parth
>>> 
>>> 12 non-binding +1s
>>> + Khurram
>>> + Dechang
>>> + Rahul
>>> + Chunhui
>>> + Karthikeyan
>>> + Robert
>>> + Paul
>>> + Krystal
>>> + Sorabh
>>> + Abhishek
>>> + Kunal
>>> + Gautam
>>> 
>>> No 0s or -1s
>>> 
>>> I'll push the release artifacts, and send an announcement once
>> propagated. Thanks to everyone involved!
>>> 
>>> Thank you,
>>> Sudheesh
>>> 
>>>> On Nov 16, 2016, at 6:23 PM, Gautam Parai <gpa...@maprtech.com> wrote:
>>>> 
>>>> +1 (non-binding)
>>>> 
>>>> Built from source on Linux VM and Mac.
>>>> Ran unit tests.
>>>> Ran new tests derived from bugs (Drill-4986/Drill-4771/Drill-
>>>> 4792/Drill-4927)
>>>> Ran some random queries
>>>> 
>>>> Found a potential bug (NON-blocker) in Drill-4792.
>>>> 
>>>> LGTM
>>>> 
>>>> On Wed, Nov 16, 2016 at 5:52 PM, Kunal Khatua <kkha...@maprtech.com>
>> wrote:
>>>> 
>>>>> +1 (non-binding)
>>>>> 
>>>>> Built from the GitHub repo and deployed on a 10-node setup.
>>>>> Ran a bunch of queries and verified the profiles as well.
>>>>> 
>>>>> LGTM.
>>>>> 
>>>>> 
>>>>> On Wed 16-Nov-2016 3:41:03 PM, Abhishek Girish <agir...@mapr.com>
>> wrote:
>>>>> +1 (non-binding)
>>>>> 
>>>>> Built from source. Ran Functional and Advanced tests from [1]. Sanity
>>>>> tested Sqlline and Web UI. Looks good.
>>>>> 
>>>>> 
>>>>> [1] https://github.com/mapr/drill-test-framework.git
>>>>> 
>>>&

Re: [ANNOUNCE] - New Apache Drill Committer - Neeraja Rentachintala

[Sudheesh Katkam]

Congratulations Neeraja!

> On Nov 18, 2016, at 2:55 AM, Tugdual Grall  wrote:
> 
> Congrats Neeraja!
> 
> t
> 
> On Thu, Nov 17, 2016 at 9:21 PM, Chunhui Shi  wrote:
> 
>> Congratulations!
>> 
>> 
>> On Thu, Nov 17, 2016 at 11:23 AM, Subbu Srinivasan <
>> ssriniva...@zscaler.com>
>> wrote:
>> 
>>> Congrats.
>>> Wish I had more time to contribute more(, perils of working in a
>>> startup)
>>> 
>>> 
>>> On Thu, Nov 17, 2016 at 11:18 AM, Jason Altekruse 
>>> wrote:
>>> 
 Congratulations! Thanks for all your contributions to Drill!
 
 Jason Altekruse
 Software Engineer at Dremio
 Apache Drill Committer
 
 On Thu, Nov 17, 2016 at 11:12 AM, Abhishek Girish 
 wrote:
 
> Congrats Neeraja!
> 
> On Thu, Nov 17, 2016 at 11:10 AM, Parth Chandra 
 wrote:
> 
>> On behalf of the Apache Drill PMC, I am very pleased to announce
>> that
>> Neeraja Rentachintala has accepted the invitation to become a
>>> committer
> in
>> the project.
>> 
>> 
>> Welcome Neeraja !
>> 
> 
 
>>> 
>> 

Re: [RESULT] [VOTE] Release Apache Drill 1.9.0 RC1

[Sudheesh Katkam]

Hi all,

I had not noticed that Gautam mentioned about a potential bug. That is a -1 
from me on the proposed candidate; the bug is a regression in behavior. I did 
not push the release artifacts until now, and the announcement is not out.

The issue is that the query profile is not displayed past the point of failure 
(trying to show a changed string option). So I will propose another candidate 
once this issue is fixed [1, 2].

In the mean time, please test the candidate for other regressions.

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-5047 
<https://issues.apache.org/jira/browse/DRILL-5047>
[2] https://github.com/apache/drill/pull/655 
<https://github.com/apache/drill/pull/655>

> On Nov 16, 2016, at 7:15 PM, Sudheesh Katkam <skat...@maprtech.com> wrote:
> 
> The proposal passes!
> 
> Final tally:
> 
> 3 binding +1s
> + Sudheesh
> + Aman
> + Parth
> 
> 12 non-binding +1s
> + Khurram
> + Dechang
> + Rahul
> + Chunhui
> + Karthikeyan
> + Robert
> + Paul
> + Krystal
> + Sorabh
> + Abhishek
> + Kunal
> + Gautam
> 
> No 0s or -1s
> 
> I'll push the release artifacts, and send an announcement once propagated. 
> Thanks to everyone involved!
> 
> Thank you,
> Sudheesh
> 
>> On Nov 16, 2016, at 6:23 PM, Gautam Parai <gpa...@maprtech.com> wrote:
>> 
>> +1 (non-binding)
>> 
>> Built from source on Linux VM and Mac.
>> Ran unit tests.
>> Ran new tests derived from bugs (Drill-4986/Drill-4771/Drill-
>> 4792/Drill-4927)
>> Ran some random queries
>> 
>> Found a potential bug (NON-blocker) in Drill-4792.
>> 
>> LGTM
>> 
>> On Wed, Nov 16, 2016 at 5:52 PM, Kunal Khatua <kkha...@maprtech.com> wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> Built from the GitHub repo and deployed on a 10-node setup.
>>> Ran a bunch of queries and verified the profiles as well.
>>> 
>>> LGTM.
>>> 
>>> 
>>> On Wed 16-Nov-2016 3:41:03 PM, Abhishek Girish <agir...@mapr.com> wrote:
>>> +1 (non-binding)
>>> 
>>> Built from source. Ran Functional and Advanced tests from [1]. Sanity
>>> tested Sqlline and Web UI. Looks good.
>>> 
>>> 
>>> [1] https://github.com/mapr/drill-test-framework.git
>>> 
>>> 
>>> On Wed, Nov 16, 2016 at 3:37 PM, Sorabh Hamirwasia
>>>> wrote:
>>> 
>>>> +1 (non-binding)
>>>> Built from source and successfully ran unit tests.
>>>> Ran both in embedded and distributed mode.
>>>> Verified DRILL-4972 / DRILL-4964
>>>> Ran some basic query on sys tables and sample data.
>>>> 
>>>> Looks good.
>>>> 
>>>> 
>>>> On Wed, Nov 16, 2016 at 2:49 PM, Krystal Nguyen
>>>> wrote:
>>>> 
>>>>> +1 (non-binding)
>>>>> Built from source. Tested the WebUI including authentication. Tested
>>>>> sqlline.
>>>>> 
>>>>> On Wed, Nov 16, 2016 at 1:59 PM, Paul Rogers
>>>> wrote:
>>>>> 
>>>>>> +1 (non-binding)
>>>>>> Built from source
>>>>>> Ran script unit tests to verify config settings, etc.
>>>>>> 
>>>>>> Looks good.
>>>>>> 
>>>>>> - Paul
>>>>>> 
>>>>>>> On Nov 16, 2016, at 1:46 PM, Robert Hou wrote:
>>>>>>> 
>>>>>>> +1 (non-binding)
>>>>>>> 
>>>>>>> Built from source.
>>>>>>> Tested parquet filter pushdown.
>>>>>>> 
>>>>>>> On Wed, Nov 16, 2016 at 1:22 PM, Karthikeyan Manivannan
>>>>>>> kmanivan...@maprtech.com> wrote:
>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> Built from source.
>>>>>>>> Ran tests in embedded mode to verify the fix for DRILL-4974.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Wed, Nov 16, 2016 at 1:11 PM, Parth Chandra
>>>> pchan...@maprtech.com
>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> +1 (binding)
>>>>>>>>> 
>>>>>>>>> Checked the source and binary signatures.
>>>>>>>>> Built from source.
>>>>>>>>> Ran a few hundred queries agai

Re: [VOTE] Release Apache Drill 1.9.0 RC1

[Sudheesh Katkam]

Hi all,

The vote ends tomorrow at 6:30 PM PT; please vote!

As of now, there are only two binding votes.

Thank you,
Sudheesh

> On Nov 14, 2016, at 7:51 AM, Dechang Gu <d...@maprtech.com> wrote:
> 
> +1
> 
> - build from source
> - deployed on a cluster
> - run TPCH and TPCDS SF100
> 
> LGTM.
> 
> -Dechang
> 
> On Sun, Nov 13, 2016 at 6:13 PM, Aman Sinha <amansi...@apache.org> wrote:
> 
>> +1 (binding)
>> 
>> - Downloaded the binaries on my mac, verified README, git.properties and
>> KEYS file GPG key
>> - Ran several queries, including CTAS against TPC-H data.  Checked Explain
>> plans and results for a few queries.
>> - Checked Web UI for query profiles.
>> - Downloaded source on my Linux VM, did a build and ran unit tests
>> successfully.
>> - Checked Maven artifacts on repositories.apache.org
>> 
>> -Aman
>> 
>> 
>> On Sat, Nov 12, 2016 at 12:27 PM, Khurram Faraaz <kfar...@maprtech.com>
>> wrote:
>> 
>>> Built from source without unit tests.
>>> deployed binaries on a cluster.
>>> executed some basic SQL queries (like aggregation, joins, range search
>> etc)
>>> from sqlline.
>>> 
>>> looks good to me.
>>> 
>>> On Sat, Nov 12, 2016 at 7:48 AM, Sudheesh Katkam <sudhe...@apache.org>
>>> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I would like to propose the second release candidate (RC1) of Apache
>>> Drill,
>>>> version 1.9.0. Thanks to everyone who contributed to this release!
>>>> 
>>>> + Compared to RC0, this release candidate does not contain DRILL-4373,
>>> due
>>>> to a regression (DRILL-5034).
>>>> + The release candidate covers a total of 73 resolved JIRAs [1].
>>>> + The tarball artifacts are hosted at [2], and the maven artifacts are
>>>> hosted at [3].
>>>> + This release candidate is based on commit
>>>> db3085498c2dc481f734733535c877dfffb9afea located at [4].
>>>> + The artifacts are signed with the key at [5].
>>>> 
>>>> The vote ends at 6:30 PM PT, November 16th, 2016.
>>>> 
>>>> Here's my vote: +1
>>>> 
>>>> Thank you,
>>>> Sudheesh
>>>> 
>>>> [1]
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>>> projectId=12313820=12337861
>>>> [2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc1/
>>>> [3] https://repository.apache.org/content/repositories/
>>>> orgapachedrill-1038/
>>>> [4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
>>>> [5] https://people.apache.org/keys/committer/sudheesh.asc
>>>> 
>>> 
>> 

[VOTE] Release Apache Drill 1.9.0 RC1

[Sudheesh Katkam]

Hi all,

I would like to propose the second release candidate (RC1) of Apache Drill,
version 1.9.0. Thanks to everyone who contributed to this release!

+ Compared to RC0, this release candidate does not contain DRILL-4373, due
to a regression (DRILL-5034).
+ The release candidate covers a total of 73 resolved JIRAs [1].
+ The tarball artifacts are hosted at [2], and the maven artifacts are
hosted at [3].
+ This release candidate is based on commit
db3085498c2dc481f734733535c877dfffb9afea located at [4].
+ The artifacts are signed with the key at [5].

The vote ends at 6:30 PM PT, November 16th, 2016.

Here's my vote: +1

Thank you,
Sudheesh

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12337861
[2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc1/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1038/
[4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
[5] https://people.apache.org/keys/committer/sudheesh.asc

Re: [VOTE] Release Apache Drill 1.9.0 RC0

[Sudheesh Katkam]

Given the comments from Vitalii on DRILL-5034 and DRILL-4373, I will revert 
DRILL-4373 from 1.9 release since the issue requires a detailed discussion.

I will not back out the change from master branch, and assign the issue to be 
fixed in the next release. I will send out a new release candidate soon.

Thank you,
Sudheesh

> On Nov 10, 2016, at 2:02 PM, Sudheesh Katkam <skat...@maprtech.com> wrote:
> 
> Okay, that sinks the release. Let’s wait for a fix from the author. I will 
> propose another candidate once this issue is fixed.
> 
> In the mean time, please test the candidate for other regressions.
> 
> Thank you,
> Sudheesh
> 
>> On Nov 10, 2016, at 1:48 PM, Krystal Nguyen <kngu...@maprtech.com> wrote:
>> 
>> Found a regression - https://issues.apache.org/jira/browse/DRILL-5034
>> 
>> so -1 from me
>> 
>> On Wed, Nov 9, 2016 at 3:57 PM, Sudheesh Katkam <sudhe...@apache.org> wrote:
>> 
>>> Hi all,
>>> 
>>> I would like to propose the first release candidate (rc0) of Apache Drill,
>>> version 1.9.0. Thanks to everyone who contributed to this release!
>>> 
>>> + The release candidate covers a total of 74 resolved JIRAs [1].
>>> + The tarball artifacts are hosted at [2], and the maven artifacts are
>>> hosted at [3].
>>> + This release candidate is based on commit
>>> 5cea9afa6278e21574c6a982ae5c3d82085ef904 located at [4].
>>> + The artifacts are signed with the key at [5].
>>> 
>>> The vote ends at 4:00 PM PT, November 14th, 2016.
>>> 
>>> Here's my vote: +1
>>> 
>>> Thank you,
>>> Sudheesh
>>> 
>>> [1]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12313820=12337861
>>> [2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc0/
>>> [3] https://repository.apache.org/content/repositories/
>>> orgapachedrill-1037/
>>> [4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
>>> [5] https://people.apache.org/keys/committer/sudheesh.asc
>>> 
> 

[VOTE] Release Apache Drill 1.9.0 RC0

[Sudheesh Katkam]

Hi all,

I would like to propose the first release candidate (rc0) of Apache Drill,
version 1.9.0. Thanks to everyone who contributed to this release!

+ The release candidate covers a total of 74 resolved JIRAs [1].
+ The tarball artifacts are hosted at [2], and the maven artifacts are
hosted at [3].
+ This release candidate is based on commit
5cea9afa6278e21574c6a982ae5c3d82085ef904 located at [4].
+ The artifacts are signed with the key at [5].

The vote ends at 4:00 PM PT, November 14th, 2016.

Here's my vote: +1

Thank you,
Sudheesh

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820=12337861
[2] http://people.apache.org/~sudheesh/drill/releases/1.9.0/rc0/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1037/
[4] https://github.com/sudheeshkatkam/drill/commits/drill-1.9.0
[5] https://people.apache.org/keys/committer/sudheesh.asc

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

I am currently in the process of committing DRILL-5009 and DRILL-5007.

I will send out a release candidate after that.

Thank you,
Sudheesh

> On Nov 7, 2016, at 2:14 PM, Sudheesh Katkam <skat...@maprtech.com> wrote:
> 
> Yes, I meant DRILL-4730; let’s resolve the ticket after the release when we 
> have more time to review.
> 
> Currently, rc0 is held up by a regression [1].
> 
> Thank you,
> Sudheesh
> 
> [1] https://issues.apache.org/jira/browse/DRILL-5009 
> <https://issues.apache.org/jira/browse/DRILL-5009>
> 
>> On Nov 4, 2016, at 9:56 PM, Laurent Goujon <laur...@dremio.com 
>> <mailto:laur...@dremio.com>> wrote:
>> 
>> I guess it's DRILL-4730 and not DRILL-4370
>> 
>> On Fri, Nov 4, 2016 at 6:23 PM, Sudheesh Katkam <sudhe...@apache.org 
>> <mailto:sudhe...@apache.org>> wrote:
>> 
>>> Out of the 17 requested tickets, we resolved 13 over the week, and 4 have
>>> been deferred (DRILL-4280, DRILL-4858, DRILL-4370, DRILL-4706). Thank you
>>> everyone!
>>> 
>>> I get will get the RC0 out on Monday.
>>> 
>>> - Sudheesh
>>> 
>>> On Fri, Nov 4, 2016 at 11:49 AM, Jinfeng Ni <j...@apache.org 
>>> <mailto:j...@apache.org>> wrote:
>>> 
>>>> Agreed with Parth that we probably should start a separate thread to
>>>> discuss release version number after 1.9.0.
>>>> 
>>>> I'll start a new thread to discuss that, and leave this thread for
>>>> drill 1.9.0 release matters.
>>>> 
>>>> 
>>>> On Thu, Nov 3, 2016 at 3:53 PM, Sudheesh Katkam <sudhe...@apache.org 
>>>> <mailto:sudhe...@apache.org>>
>>>> wrote:
>>>>> Gentle reminder that all check-ins should be done by tomorrow. Please
>>> see
>>>>> the latest statuses of commits that we are targeting:
>>>>> 
>>>>> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_ 
>>>>> <https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_>
>>>>> JzkwAcXSxmcbG7meBDad6ZTxlSmw
>>>>> 
>>>>> Thank you,
>>>>> Sudheesh
>>>>> 
>>>>> 
>>>>> On Tue, Nov 1, 2016 at 11:19 AM, Sudheesh Katkam <sudhe...@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> The current list of candidate commits for the release is here:
>>>>>> 
>>>>>> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
>>>>>> JzkwAcXSxmcbG7meBDad6ZTxlSmw
>>>>>> 
>>>>>> 
>>>>>> On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <
>>>> ssriniva...@zscaler.com <mailto:ssriniva...@zscaler.com>
>>>>>>> wrote:
>>>>>> 
>>>>>>> +1.
>>>>>>> 
>>>>>>> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers <prog...@maprtech.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good
>>>> idea.
>>>>>>>> 
>>>>>>>> At first it may seem odd to go to 1.10 from 1.9. Might people get
>>>>>>> confused
>>>>>>>> between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
>>>>>>> 7-series
>>>>>>>> release is 7.0.72. Java is on 8u112. And so on.
>>>>>>>> 
>>>>>>>> I like the idea of moving to 2.0 later when the team introduces a
>>>> major
>>>>>>>> change, rather than by default just because the numbers roll
>>> around.
>>>> For
>>>>>>>> example, Hadoop when to 2.x when YARN was introduced. Impala
>>> appears
>>>> to
>>>>>>>> have moved to 2.0 when they added Spill to disk for some (all?)
>>>>>>> operators.
>>>>>>>> 
>>>>>>>> - Paul
>>>>>>>> 
>>>>>>>>> On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <
>>> sudhe...@apache.org <mailto:sudhe...@apache.org>
>>>>> 
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Drillers,
>>>>>>>>> 
>>>>>>>>> We have a reasonable number of fixes and features since the last
>>>>>>> release
>>>>>>>>> [1]. Releasing itself takes a while; so I propose we start the
>>> 1.9
>>>>>>>> release
>>>>>>>>> process.
>>>>>>>>> 
>>>>>>>>> I volunteer as the release manager, unless there are objections.
>>>>>>>>> 
>>>>>>>>> We should also discuss what the release version number should be
>>>> after
>>>>>>>> 1.9.
>>>>>>>>> 
>>>>>>>>> Thank you,
>>>>>>>>> Sudheesh
>>>>>>>>> 
>>>>>>>>> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/
>>>>>>> 12337861
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
> 

Re: Proposed November report for Drill

[Sudheesh Katkam]

+1

> On Nov 7, 2016, at 10:35 PM, Parth Chandra <par...@apache.org> wrote:
> 
> Hi Drill devs,
> 
> Below is the proposed report for the November Apache board meeting. Please
> provide any comments/corrections. I'll be submitting this Wednesday Nov
> 9th, so please provide input before that.
> 
> Thanks
> 
> Parth
> 
> --- Report Begin -
> 
> ## Description:
> - Drill is a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud
> Storage
> 
> ## Issues:
> 
> - There are no issues requiring board attention at this time
> 
> ## Activity:
> 
> - Since the last board report, Drill has released version 1.8
> - Drill has added many new features referred to in the last report.
> Dynamic UDFs, Parquet reader performance improvements, filter pushdown for
> Parquet, and improved support for Metadata in the clients has been added.
> - Improved use of statistics, and security enhancements (including support
> for Kerberos) continue to be in the works. Also in progress is an
> improvement to the data locality algorithm.
> 
> ## Health report:
> 
> - There has been a good increase in the number posts in the dev and jira
> lists. This reflects the increased activity on the development front. User
> list activity is down this period, but not a concern at the moment.
> 
> ## PMC changes:
> 
> - Currently 18 PMC members.
> - Sudheesh Katkam was added to the PMC on Wed Oct 05 2016
> 
> ## Committer base changes:
> 
> - Currently 28 committers.
> - No new committers added in the last 3 months
> - Last committer addition was Hsuan-Yi Chu at Thu Apr 07 2016
> 
> ## Releases:
> 
> - 1.8.0 was released on Mon Aug 29 2016
> 
> ## Mailing list activity:
> 
> - dev@drill.apache.org:
>- 436 subscribers (down -10 in the last 3 months):
>- 1797 emails sent to list (1231 in previous quarter)
> 
> - iss...@drill.apache.org:
>- 20 subscribers (up 0 in the last 3 months):
>- 2188 emails sent to list (1550 in previous quarter)
> 
> - u...@drill.apache.org:
>- 567 subscribers (down -14 in the last 3 months):
>- 436 emails sent to list (824 in previous quarter)
> 
> ## JIRA activity:
> 
> - 173 JIRA tickets created in the last 3 months
> - 88 JIRA tickets closed/resolved in the last 3 months
> 
> --- Report End -

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

Yes, I meant DRILL-4730; let’s resolve the ticket after the release when we 
have more time to review.

Currently, rc0 is held up by a regression [1].

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-5009 
<https://issues.apache.org/jira/browse/DRILL-5009>

> On Nov 4, 2016, at 9:56 PM, Laurent Goujon <laur...@dremio.com> wrote:
> 
> I guess it's DRILL-4730 and not DRILL-4370
> 
> On Fri, Nov 4, 2016 at 6:23 PM, Sudheesh Katkam <sudhe...@apache.org> wrote:
> 
>> Out of the 17 requested tickets, we resolved 13 over the week, and 4 have
>> been deferred (DRILL-4280, DRILL-4858, DRILL-4370, DRILL-4706). Thank you
>> everyone!
>> 
>> I get will get the RC0 out on Monday.
>> 
>> - Sudheesh
>> 
>> On Fri, Nov 4, 2016 at 11:49 AM, Jinfeng Ni <j...@apache.org> wrote:
>> 
>>> Agreed with Parth that we probably should start a separate thread to
>>> discuss release version number after 1.9.0.
>>> 
>>> I'll start a new thread to discuss that, and leave this thread for
>>> drill 1.9.0 release matters.
>>> 
>>> 
>>> On Thu, Nov 3, 2016 at 3:53 PM, Sudheesh Katkam <sudhe...@apache.org>
>>> wrote:
>>>> Gentle reminder that all check-ins should be done by tomorrow. Please
>> see
>>>> the latest statuses of commits that we are targeting:
>>>> 
>>>> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
>>>> JzkwAcXSxmcbG7meBDad6ZTxlSmw
>>>> 
>>>> Thank you,
>>>> Sudheesh
>>>> 
>>>> 
>>>> On Tue, Nov 1, 2016 at 11:19 AM, Sudheesh Katkam <sudhe...@apache.org>
>>>> wrote:
>>>> 
>>>>> The current list of candidate commits for the release is here:
>>>>> 
>>>>> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
>>>>> JzkwAcXSxmcbG7meBDad6ZTxlSmw
>>>>> 
>>>>> 
>>>>> On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <
>>> ssriniva...@zscaler.com
>>>>>> wrote:
>>>>> 
>>>>>> +1.
>>>>>> 
>>>>>> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers <prog...@maprtech.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good
>>> idea.
>>>>>>> 
>>>>>>> At first it may seem odd to go to 1.10 from 1.9. Might people get
>>>>>> confused
>>>>>>> between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
>>>>>> 7-series
>>>>>>> release is 7.0.72. Java is on 8u112. And so on.
>>>>>>> 
>>>>>>> I like the idea of moving to 2.0 later when the team introduces a
>>> major
>>>>>>> change, rather than by default just because the numbers roll
>> around.
>>> For
>>>>>>> example, Hadoop when to 2.x when YARN was introduced. Impala
>> appears
>>> to
>>>>>>> have moved to 2.0 when they added Spill to disk for some (all?)
>>>>>> operators.
>>>>>>> 
>>>>>>> - Paul
>>>>>>> 
>>>>>>>> On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <
>> sudhe...@apache.org
>>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Drillers,
>>>>>>>> 
>>>>>>>> We have a reasonable number of fixes and features since the last
>>>>>> release
>>>>>>>> [1]. Releasing itself takes a while; so I propose we start the
>> 1.9
>>>>>>> release
>>>>>>>> process.
>>>>>>>> 
>>>>>>>> I volunteer as the release manager, unless there are objections.
>>>>>>>> 
>>>>>>>> We should also discuss what the release version number should be
>>> after
>>>>>>> 1.9.
>>>>>>>> 
>>>>>>>> Thank you,
>>>>>>>> Sudheesh
>>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/
>>>>>> 12337861
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

Out of the 17 requested tickets, we resolved 13 over the week, and 4 have
been deferred (DRILL-4280, DRILL-4858, DRILL-4370, DRILL-4706). Thank you
everyone!

I get will get the RC0 out on Monday.

- Sudheesh

On Fri, Nov 4, 2016 at 11:49 AM, Jinfeng Ni <j...@apache.org> wrote:

> Agreed with Parth that we probably should start a separate thread to
> discuss release version number after 1.9.0.
>
> I'll start a new thread to discuss that, and leave this thread for
> drill 1.9.0 release matters.
>
>
> On Thu, Nov 3, 2016 at 3:53 PM, Sudheesh Katkam <sudhe...@apache.org>
> wrote:
> > Gentle reminder that all check-ins should be done by tomorrow. Please see
> > the latest statuses of commits that we are targeting:
> >
> > https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
> > JzkwAcXSxmcbG7meBDad6ZTxlSmw
> >
> > Thank you,
> > Sudheesh
> >
> >
> > On Tue, Nov 1, 2016 at 11:19 AM, Sudheesh Katkam <sudhe...@apache.org>
> > wrote:
> >
> >> The current list of candidate commits for the release is here:
> >>
> >> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
> >> JzkwAcXSxmcbG7meBDad6ZTxlSmw
> >>
> >>
> >> On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <
> ssriniva...@zscaler.com
> >> > wrote:
> >>
> >>> +1.
> >>>
> >>> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers <prog...@maprtech.com>
> >>> wrote:
> >>>
> >>> > For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good
> idea.
> >>> >
> >>> > At first it may seem odd to go to 1.10 from 1.9. Might people get
> >>> confused
> >>> > between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
> >>> 7-series
> >>> > release is 7.0.72. Java is on 8u112. And so on.
> >>> >
> >>> > I like the idea of moving to 2.0 later when the team introduces a
> major
> >>> > change, rather than by default just because the numbers roll around.
> For
> >>> > example, Hadoop when to 2.x when YARN was introduced. Impala appears
> to
> >>> > have moved to 2.0 when they added Spill to disk for some (all?)
> >>> operators.
> >>> >
> >>> > - Paul
> >>> >
> >>> > > On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <sudhe...@apache.org
> >
> >>> > wrote:
> >>> > >
> >>> > > Hi Drillers,
> >>> > >
> >>> > > We have a reasonable number of fixes and features since the last
> >>> release
> >>> > > [1]. Releasing itself takes a while; so I propose we start the 1.9
> >>> > release
> >>> > > process.
> >>> > >
> >>> > > I volunteer as the release manager, unless there are objections.
> >>> > >
> >>> > > We should also discuss what the release version number should be
> after
> >>> > 1.9.
> >>> > >
> >>> > > Thank you,
> >>> > > Sudheesh
> >>> > >
> >>> > > [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/
> >>> 12337861
> >>> >
> >>> >
> >>>
> >>
> >>
>

[jira] [Resolved] (DRILL-4969) Basic implementation for displaySize

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4969.

Resolution: Fixed
  Reviewer: Sudheesh Katkam

Fixed in 
[3fd6938|https://github.com/apache/drill/commit/3fd69387591ebd56286b39749adc1dd721b3f14d]

> Basic implementation for displaySize
> 
>
> Key: DRILL-4969
> URL: https://issues.apache.org/jira/browse/DRILL-4969
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
> Fix For: 1.9.0
>
>
> display size is fixed to 10, but for most types display size is well defined 
> as shown in the ODBC table:
> https://msdn.microsoft.com/en-us/library/ms713974(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4945) Missing subtype information in metadata returned by prepared statement

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4945.

   Resolution: Fixed
 Assignee: Laurent Goujon
 Reviewer: Sudheesh Katkam
Fix Version/s: 1.9.0

Fixed in 
[2365ac0|https://github.com/apache/drill/commit/2365ac05524e21b2d084e5138e70a24092fe1bd8]

> Missing subtype information in metadata returned by prepared statement
> --
>
> Key: DRILL-4945
> URL: https://issues.apache.org/jira/browse/DRILL-4945
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
>Priority: Minor
> Fix For: 1.9.0
>
>
> Column metadata returned by prepared statement contains partial type 
> information, especially for interval types. 
> Currently it only returns "INTERVAL" instead of a more precise type like 
> "INTERVAL MONTH" for example.
> There's also no minor type, so the client can adjust the type itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

Gentle reminder that all check-ins should be done by tomorrow. Please see
the latest statuses of commits that we are targeting:

https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
JzkwAcXSxmcbG7meBDad6ZTxlSmw

Thank you,
Sudheesh


On Tue, Nov 1, 2016 at 11:19 AM, Sudheesh Katkam <sudhe...@apache.org>
wrote:

> The current list of candidate commits for the release is here:
>
> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
> JzkwAcXSxmcbG7meBDad6ZTxlSmw
>
>
> On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <ssriniva...@zscaler.com
> > wrote:
>
>> +1.
>>
>> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers <prog...@maprtech.com>
>> wrote:
>>
>> > For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good idea.
>> >
>> > At first it may seem odd to go to 1.10 from 1.9. Might people get
>> confused
>> > between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
>> 7-series
>> > release is 7.0.72. Java is on 8u112. And so on.
>> >
>> > I like the idea of moving to 2.0 later when the team introduces a major
>> > change, rather than by default just because the numbers roll around. For
>> > example, Hadoop when to 2.x when YARN was introduced. Impala appears to
>> > have moved to 2.0 when they added Spill to disk for some (all?)
>> operators.
>> >
>> > - Paul
>> >
>> > > On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <sudhe...@apache.org>
>> > wrote:
>> > >
>> > > Hi Drillers,
>> > >
>> > > We have a reasonable number of fixes and features since the last
>> release
>> > > [1]. Releasing itself takes a while; so I propose we start the 1.9
>> > release
>> > > process.
>> > >
>> > > I volunteer as the release manager, unless there are objections.
>> > >
>> > > We should also discuss what the release version number should be after
>> > 1.9.
>> > >
>> > > Thank you,
>> > > Sudheesh
>> > >
>> > > [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/
>> 12337861
>> >
>> >
>>
>
>

[jira] [Resolved] (DRILL-4986) Allow users to customize the Drill log file name

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4986.

Resolution: Fixed
  Assignee: Paul Rogers
  Reviewer: Boaz Ben-Zvi

Fixed in 
[e044f5a|https://github.com/apache/drill/commit/e044f5a9d007266b9094fb8899d0a05df0b0dd0d]

> Allow users to customize the Drill log file name
> 
>
> Key: DRILL-4986
> URL: https://issues.apache.org/jira/browse/DRILL-4986
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.9.0
>
>
> From John Omernik on the user list:
> I am getting my head around the new Drill config files in 1.8, and I have
> been setting DRILL_LOG_PREFIX in previous versions. Based on what I am
> seeing in drill-config.sh though, it looks like I no longer can set this as
> instead of using a "If set use what's set otherwise default to" methodology
> like other variables, drill-config.sh just exports a static value... is
> this correct?
> My goal is to have logs all in a single directory but have prefixes based
> on the host names... "centralized" logging if you will :)
> Suggestion:
> {code}
> export DRILL_LOG_PREFIX="$DRILL_LOG_DIR/drillbit"
> {code}
> So looking at this, I think that DRILL_LOG_PREFIX is a misnomer, it should
> be DRILL_LOG_PATH = Path to a directory, DRILL_LOG_PREFIX = A prefix to
> prepend to drill log files, thus, in reality, DRILL_LOG_PREFIX should not
> be set to $DRILL_LOG_DIR/drillbit, instead the default should be "
> {code}
> export DRILL_LOG_PREFIX=${DRILL_LOG_PREFIX:-"drillbit"}
> {code}
> and then the next line that makes the drill log path should be:
> {code}
> export DRILLBIT_LOG_PATH="${DRILL_LOG_DIR}/${DRILL_LOG_PREFIX}.log"
> {code}
> That way the Prefix is just a file prefix, and then in the drill-env.sh I
> could set
> {code}
> export DRILL_LOG_PREFIX="drillbit-$HOSTNAME"
> {code}
> and have all my logs in one folder, but not be overwritten by easy other



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-1268) C++ Client. Write Unit Test for Drill Client

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-1268.

   Resolution: Fixed
 Assignee: Laurent Goujon
 Reviewer: Parth Chandra
Fix Version/s: (was: Future)
   1.9.0

Fixed in 
[3a35a42|https://github.com/apache/drill/commit/3a35a4200e748ed557c55d6d13ac995cce28ab09]

> C++ Client. Write Unit Test for Drill Client
> 
>
> Key: DRILL-1268
> URL: https://issues.apache.org/jira/browse/DRILL-1268
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Client - C++
>Reporter: Xiao Meng
>Assignee: Laurent Goujon
> Fix For: 1.9.0
>
>
> Currently, C++ client uses an example program QuerySubmitter for testing, we 
> should write unit tests for C++ client. 
> Candidate unit test framework:
> - [Google Test|https://code.google.com/p/googletest/]
> - Boost Test
> - [Catch|https://github.com/philsquared/Catch] 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4853) Update C++ protobuf source files

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4853.

   Resolution: Fixed
 Reviewer: Parth Chandra
Fix Version/s: 1.9.0

Fixed in 
[2558803|https://github.com/apache/drill/commit/2558803ecdfc961bb630e9e2372255c44f986d06]

> Update C++ protobuf source files
> 
>
> Key: DRILL-4853
> URL: https://issues.apache.org/jira/browse/DRILL-4853
> Project: Apache Drill
>  Issue Type: Task
>  Components: Client - C++
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
> Fix For: 1.9.0
>
>
> C++ protobuf files have not been updated since january, missing changes for 
> prepared statement and metadata querying support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3423) Add New HTTPD format plugin

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-3423.

   Resolution: Fixed
Fix Version/s: (was: Future)
   1.9.0

Fixed in 
[818f945|https://github.com/apache/drill/commit/818f94507b9d1792282a4bb8abd2d22d2fc2eb57]

> Add New HTTPD format plugin
> ---
>
> Key: DRILL-3423
> URL: https://issues.apache.org/jira/browse/DRILL-3423
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Reporter: Jacques Nadeau
>Assignee: Jim Scott
> Fix For: 1.9.0
>
>
> Add an HTTPD logparser based format plugin.  The author has been kind enough 
> to move the logparser project to be released under the Apache License.  Can 
> find it here:
> 
> nl.basjes.parse.httpdlog
> httpdlog-parser
> 2.0
> 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Drill Hangout Minutes - 11/1/16

[Sudheesh Katkam]

Hi Laurent,

That's right; this was mentioned in the design document.

I am piggybacking on previous changes that break the "newer clients talking
to older servers" compatibility. For example, as I understand, some
resolved sub-tasks of DRILL-4714 [1] *implicitly* break this compatibility;
say the "newer" API that was introduced is used by an application which is
talking to an older server. The older server drops the connection, unable
to handle the message.

In DRILL-4280, there is an *explicit* break in that specific compatibility,
and the error message is much cleaner with a version mismatch message. The
difference is that the C++ client (unlike the Java client) checks for the
server version as well, which make the compatibility break more visible.

I am not sure about the plan of action in general about this compatibility.
However, I could work around the issue by advertising clients' SASL
capability to the server. What do you think?

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-4714

On Nov 1, 2016, at 7:49 PM, Laurent Goujon  wrote:

Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan
to defer this change to a later release, or to defer bringing back
compatibility between newer clients and older servers to a later release?

Laurent

On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong  wrote:

Oops, mistake in my notes.  For the second item, I meant DRILL-4280, not
DRILL-1950.

On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong  wrote:

Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
Arina, Laurent, Kunal, Zelaine

I had to leave the hangout at 10:30, so my notes only cover the

discussion

up till then.

1) Variable width decimal support - Dave O

Currently Drill only supports fixed width byte array storage of decimals.
Dave has submitted a pull request for DRILL-4834 to add support for

storing

decimals with variable width byte arrays.  Eventually, variable width can
replace fixed width, but the pull request doesn't cover that.  Dave would
like someone in the community to review his pull request.

2) 1.9 release - Sudheesh

Sudheesh is collecting pull requests for the release.  Some have been
reviewed and are waiting to be merged.  Sudheesh plans to commit a batch
this Wed and another this Friday.  He's targeting having a release
candidate build available next Monday.

Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
whether thought had been given to supporting newer Drill clients with

older

Drill servers.  Sudheesh indicated that doing this would entail a

breaking

change in the protocol, and the plan was to defer doing this for a later
release where we may want to make other breaking changes like this.

[jira] [Created] (DRILL-4987) Use ImpersonationUtil in RemoteFunctionRegistry

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4987:
--

 Summary: Use ImpersonationUtil in RemoteFunctionRegistry
 Key: DRILL-4987
 URL: https://issues.apache.org/jira/browse/DRILL-4987
 Project: Apache Drill
  Issue Type: Improvement
  Components: Functions - Drill
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Priority: Minor
 Fix For: 1.9.0


+ Use ImpersonationUtil#getProcessUserName rather than  
UserGroupInformation#getCurrentUser#getUserName in RemoteFunctionRegistry
+ Expose process users' group info in ImpersonationUtil and use that in 
RemoteFunctionRegistry, rather than 
UserGroupInformation#getCurrentUser#getGroupNames



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

The current list of candidate commits for the release is here:

https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_JzkwAcXSxmcbG7meBDad6ZTxlSmw


On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <ssriniva...@zscaler.com>
wrote:

> +1.
>
> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers <prog...@maprtech.com>
> wrote:
>
> > For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good idea.
> >
> > At first it may seem odd to go to 1.10 from 1.9. Might people get
> confused
> > between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
> 7-series
> > release is 7.0.72. Java is on 8u112. And so on.
> >
> > I like the idea of moving to 2.0 later when the team introduces a major
> > change, rather than by default just because the numbers roll around. For
> > example, Hadoop when to 2.x when YARN was introduced. Impala appears to
> > have moved to 2.0 when they added Spill to disk for some (all?)
> operators.
> >
> > - Paul
> >
> > > On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <sudhe...@apache.org>
> > wrote:
> > >
> > > Hi Drillers,
> > >
> > > We have a reasonable number of fixes and features since the last
> release
> > > [1]. Releasing itself takes a while; so I propose we start the 1.9
> > release
> > > process.
> > >
> > > I volunteer as the release manager, unless there are objections.
> > >
> > > We should also discuss what the release version number should be after
> > 1.9.
> > >
> > > Thank you,
> > > Sudheesh
> > >
> > > [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/12337861
> >
> >
>

Re: [HANGOUT] Topics for 11/1/16

[Sudheesh Katkam]

Back to the old one.

From drill.apache.org <http://drill.apache.org/> > Community > Community 
Resources >

https://hangouts.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc 
<https://hangouts.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc>

Thank you,
Sudheesh

> On Nov 1, 2016, at 10:02 AM, Dave Oshinsky <doshin...@commvault.com> wrote:
> 
> Is the hangout link same as earlier?
> https://hangouts.google.com/hangouts/_/maprtech.com/drillbi-weeklyhangout
> 
> -Original Message-
> From: Sudheesh Katkam [mailto:sudhe...@apache.org] 
> Sent: Tuesday, November 01, 2016 12:52 PM
> To: dev
> Subject: Re: [HANGOUT] Topics for 11/1/16
> 
> Zelaine, That list is from the other thread ("Time for a 1.9 Release?"), 
> where there was no mention of PR#639.
> 
> Charles, DRILL-3423 is already on the list.
> 
> Thank you,
> Sudheesh
> 
> On Tue, Nov 1, 2016 at 9:43 AM, Charles Givre <cgi...@gmail.com> wrote:
> 
>> I think DRILL-3423 HTTPD parser should be added as well. I'm sorry I'm 
>> traveling at the moment and won't be able to join the hangout.
>> 
>> Sent from my iPhone
>> 
>>> On Nov 1, 2016, at 12:40, Zelaine Fong <zf...@maprtech.com> wrote:
>>> 
>>> Regarding [1], shouldn't https://github.com/apache/drill/pull/639 be
>> added
>>> to the list of pull requests in review?
>>> 
>>> -- Zelaine
>>> 
>>> 
>>>> On Tue, Nov 1, 2016 at 9:34 AM, Sudheesh Katkam 
>>>> <sudhe...@apache.org>
>> wrote:
>>>> 
>>>> I would like to talk about the 1.9 release. The current list of 
>>>> pending requests is at [1].
>>>> 
>>>> If Arina joins the hangout, I would like to discuss about 
>>>> permissions on the various areas setup for dynamic UDF support.
>>>> 
>>>> Thank you,
>>>> Sudheesh
>>>> 
>>>> [1]
>>>> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_JzkwA
>>>> cXSxmcbG7meBDad6ZTxlSmw
>>>> 
>>>> 
>>>> On Mon, Oct 31, 2016 at 10:03 AM, Dave Oshinsky <
>> doshin...@commvault.com>
>>>> wrote:
>>>> 
>>>>> Hi Paul,
>>>>> Can we talk a bit about working to improve decimal type support?  
>>>>> My related PR is waiting for review:
>>>>> https://issues.apache.org/jira/browse/DRILL-4834
>>>>> 
>>>>> Thanks,
>>>>> Dave Oshinsky
>>>>> 
>>>>> -Original Message-
>>>>> From: Paul Rogers [mailto:prog...@maprtech.com]
>>>>> Sent: Monday, October 31, 2016 12:33 PM
>>>>> To: dev@drill.apache.org; u...@drill.apache.org
>>>>> Subject: [HANGOUT] Topics for 11/1/16
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> Our bi-weekly hangout is tomorrow (11/01/16, 10 AM PT). Please 
>>>>> respond with suggested topics. We will also ask for additional 
>>>>> topics at the beginning of the hangout.
>>>>> 
>>>>> See you then,
>>>>> 
>>>>> - Paul
>>>>> 
>>>>> 
>>>>> ***Legal 
>>>>> Disclaimer***
>>>>> "This communication may contain confidential and privileged 
>>>>> material
>> for
>>>>> the
>>>>> sole use of the intended recipient. Any unauthorized review, use 
>>>>> or distribution by others is strictly prohibited. If you have 
>>>>> received the message by mistake, please advise the sender by reply 
>>>>> email and delete the message. Thank
>>>> you."
>>>>> **
>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for the
> sole use of the intended recipient. Any unauthorized review, use or 
> distribution
> by others is strictly prohibited. If you have received the message by mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **

Re: [HANGOUT] Topics for 11/1/16

[Sudheesh Katkam]

Zelaine, That list is from the other thread ("Time for a 1.9 Release?"),
where there was no mention of PR#639.

Charles, DRILL-3423 is already on the list.

Thank you,
Sudheesh

On Tue, Nov 1, 2016 at 9:43 AM, Charles Givre <cgi...@gmail.com> wrote:

> I think DRILL-3423 HTTPD parser should be added as well. I'm sorry I'm
> traveling at the moment and won't be able to join the hangout.
>
> Sent from my iPhone
>
> > On Nov 1, 2016, at 12:40, Zelaine Fong <zf...@maprtech.com> wrote:
> >
> > Regarding [1], shouldn't https://github.com/apache/drill/pull/639 be
> added
> > to the list of pull requests in review?
> >
> > -- Zelaine
> >
> >
> >> On Tue, Nov 1, 2016 at 9:34 AM, Sudheesh Katkam <sudhe...@apache.org>
> wrote:
> >>
> >> I would like to talk about the 1.9 release. The current list of pending
> >> requests is at [1].
> >>
> >> If Arina joins the hangout, I would like to discuss about permissions on
> >> the various areas setup for dynamic UDF support.
> >>
> >> Thank you,
> >> Sudheesh
> >>
> >> [1]
> >> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_JzkwA
> >> cXSxmcbG7meBDad6ZTxlSmw
> >>
> >>
> >> On Mon, Oct 31, 2016 at 10:03 AM, Dave Oshinsky <
> doshin...@commvault.com>
> >> wrote:
> >>
> >>> Hi Paul,
> >>> Can we talk a bit about working to improve decimal type support?  My
> >>> related PR is waiting for review:
> >>> https://issues.apache.org/jira/browse/DRILL-4834
> >>>
> >>> Thanks,
> >>> Dave Oshinsky
> >>>
> >>> -Original Message-
> >>> From: Paul Rogers [mailto:prog...@maprtech.com]
> >>> Sent: Monday, October 31, 2016 12:33 PM
> >>> To: dev@drill.apache.org; u...@drill.apache.org
> >>> Subject: [HANGOUT] Topics for 11/1/16
> >>>
> >>> Hi All,
> >>>
> >>> Our bi-weekly hangout is tomorrow (11/01/16, 10 AM PT). Please respond
> >>> with suggested topics. We will also ask for additional topics at the
> >>> beginning of the hangout.
> >>>
> >>> See you then,
> >>>
> >>> - Paul
> >>>
> >>>
> >>> ***Legal Disclaimer***
> >>> "This communication may contain confidential and privileged material
> for
> >>> the
> >>> sole use of the intended recipient. Any unauthorized review, use or
> >>> distribution
> >>> by others is strictly prohibited. If you have received the message by
> >>> mistake,
> >>> please advise the sender by reply email and delete the message. Thank
> >> you."
> >>> **
> >>>
> >>>
> >>
>

Re: [HANGOUT] Topics for 11/1/16

[Sudheesh Katkam]

I would like to talk about the 1.9 release. The current list of pending
requests is at [1].

If Arina joins the hangout, I would like to discuss about permissions on
the various areas setup for dynamic UDF support.

Thank you,
Sudheesh

[1]
https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_JzkwAcXSxmcbG7meBDad6ZTxlSmw


On Mon, Oct 31, 2016 at 10:03 AM, Dave Oshinsky 
wrote:

> Hi Paul,
> Can we talk a bit about working to improve decimal type support?  My
> related PR is waiting for review:
> https://issues.apache.org/jira/browse/DRILL-4834
>
> Thanks,
> Dave Oshinsky
>
> -Original Message-
> From: Paul Rogers [mailto:prog...@maprtech.com]
> Sent: Monday, October 31, 2016 12:33 PM
> To: dev@drill.apache.org; u...@drill.apache.org
> Subject: [HANGOUT] Topics for 11/1/16
>
> Hi All,
>
> Our bi-weekly hangout is tomorrow (11/01/16, 10 AM PT). Please respond
> with suggested topics. We will also ask for additional topics at the
> beginning of the hangout.
>
> See you then,
>
> - Paul
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **
>
>

[jira] [Resolved] (DRILL-4884) Drill produced IOB exception while querying data of 65536 limitation using non batched reader

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4884.

   Resolution: Fixed
 Assignee: (was: Jinfeng Ni)
 Reviewer: Jinfeng Ni  (was: Daniel Barclay)
Fix Version/s: 1.9.0

Fixed in 
[1e6fa00|https://github.com/apache/drill/commit/1e6fa00cd4b0b1db41614749f6d12c03f0ca7990]

> Drill produced IOB exception while querying data of 65536 limitation using 
> non batched reader
> -
>
> Key: DRILL-4884
> URL: https://issues.apache.org/jira/browse/DRILL-4884
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Query Planning & Optimization
>Affects Versions: 1.8.0
> Environment: CentOS 6.5 / JAVA 8
>Reporter: Hongze Zhang
> Fix For: 1.9.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Drill produces IOB while using a non batched scanner and limiting SQL by 
> 65536.
> SQL:
> {noformat}
> select id from xx limit 1 offset 65535
> {noformat}
> Result:
> {noformat}
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324)
>  [classes/:na]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
>  [classes/:na]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
>  [classes/:na]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [classes/:na]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_101]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_101]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.lang.IndexOutOfBoundsException: index: 131072, length: 2 
> (expected: range(0, 131072))
>   at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:175) 
> ~[classes/:4.0.27.Final]
>   at io.netty.buffer.DrillBuf.chk(DrillBuf.java:197) 
> ~[classes/:4.0.27.Final]
>   at io.netty.buffer.DrillBuf.setChar(DrillBuf.java:517) 
> ~[classes/:4.0.27.Final]
>   at 
> org.apache.drill.exec.record.selection.SelectionVector2.setIndex(SelectionVector2.java:79)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(LimitRecordBatch.java:167)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:145)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
>  ~[classes/:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerN

Re: Time for a 1.9 Release?

[Sudheesh Katkam]

Let's aim for EOD next Friday (11/04/16) to get all changes in; I will try
to get RC0 out on Monday (11/07/16).

Current list of commits:

[Sudheesh]
+ DRILL-4280: pull request being reviewed
https://github.com/apache/drill/pull/578

[Jinfeng]
+ DRILL-1950: pull request pending

Any other pull requests that developers would like to get into the release?
Please post the status too.

Thank you,
Sudheesh

On Fri, Oct 28, 2016 at 11:35 AM, Jinfeng Ni <j...@apache.org> wrote:

> +1
>
> I'm working on DRILL-1950 to support parquet row group level filter
> pruning. I plan to submit a pull request for code review in 1-2 days,
> hopefully.
>
>
>
> On Fri, Oct 28, 2016 at 11:04 AM, Aman Sinha <amansi...@apache.org> wrote:
> > +1
> >
> > On Fri, Oct 28, 2016 at 10:34 AM, Sudheesh Katkam <sudhe...@apache.org>
> > wrote:
> >
> >> Hi Drillers,
> >>
> >> We have a reasonable number of fixes and features since the last release
> >> [1]. Releasing itself takes a while; so I propose we start the 1.9
> release
> >> process.
> >>
> >> I volunteer as the release manager, unless there are objections.
> >>
> >> We should also discuss what the release version number should be after
> 1.9.
> >>
> >> Thank you,
> >> Sudheesh
> >>
> >> [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/12337861
> >>
>

[jira] [Resolved] (DRILL-4968) Add column size information to ColumnMetadata

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4968.

   Resolution: Fixed
Fix Version/s: 1.9.0

Fixed in 
[c6dbe6a|https://github.com/apache/drill/commit/c6dbe6a2f7033114d6239a4850a9b5092e684589].

> Add column size information to ColumnMetadata
> -
>
> Key: DRILL-4968
> URL: https://issues.apache.org/jira/browse/DRILL-4968
> Project: Apache Drill
>  Issue Type: Sub-task
>  Components: Metadata
>Reporter: Laurent Goujon
>Assignee: Laurent Goujon
> Fix For: 1.9.0
>
>
> Both ODBC and JDBC needs column size information for the column metadata. 
> Instead of duplicating the logic between C++ and Java (and having to keep in 
> them sync), column size should be computed on the server so that value is 
> kept consistent across clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4972) Drillbit shuts down immediately after starting if embedded web server is disabled

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4972:
--

 Summary: Drillbit shuts down immediately after starting if 
embedded web server is disabled
 Key: DRILL-4972
 URL: https://issues.apache.org/jira/browse/DRILL-4972
 Project: Apache Drill
  Issue Type: Bug
  Components:  Server
Affects Versions: 1.8.0
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Priority: Critical
 Fix For: 1.9.0


Disable embedded web server by setting "drill.exec.http.enabled" to false. Now 
when drillbit is started, it shuts down immediately after starting.

JVM exits when the only threads running are all daemon threads. Turns out all 
threads in a drillbit, other than the thread pool started by the web server, 
are daemon. So I suggest WorkManager#StatusThread be made non-daemon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4954) allTextMode in the MapRDB plugin always return nulls

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4954.

Resolution: Fixed

Fixed in 
[4efc9f2|https://github.com/apache/drill/commit/4efc9f248ef7ef4b86660a1a73a9f44662c082ba]

> allTextMode in the MapRDB plugin always return nulls
> 
>
> Key: DRILL-4954
> URL: https://issues.apache.org/jira/browse/DRILL-4954
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - MapRDB
>Affects Versions: 1.8.0
> Environment: MapRDB
>Reporter: Boaz Ben-Zvi
>Assignee: Smidth Panchamia
> Fix For: 1.9.0
>
>
> Setting the "allTextMode" option to "true" in the MapR fs plugin, like:
>   "formats": {
> "maprdb": {
>   "type": "maprdb",
>   "allTextMode": true
> }
> makes the returned results null. Here’s an example:
> << default plugin, unchanged >>
> 0: jdbc:drill:> use mfs.tpch_sf1_maprdb_json;
> +---++
> |  ok   |summary |
> +---++
> | true  | Default schema changed to [mfs1.tpch_sf1_maprdb_json]  |
> +---++
> 1 row selected (0.153 seconds)
> 0: jdbc:drill:> select typeof(N_REGIONKEY) from nation limit 1;
> +-+
> | EXPR$0  |
> +-+
> | BIGINT  |
> +-+
> 1 row selected (0.206 seconds)
> 0: jdbc:drill:> select N_REGIONKEY from nation limit 2;
> +--+
> | N_REGIONKEY  |
> +--+
> | 0|
> | 2|
> +--+
> 2 rows selected (0.254 seconds)
> << plugin changed to all text mode (as shown above) >>
> 0: jdbc:drill:> select typeof(N_REGIONKEY) from nation limit 1;
> +-+
> | EXPR$0  |
> +-+
> | NULL|
> +-+
> 1 row selected (0.321 seconds)
> 0: jdbc:drill:> select N_REGIONKEY from nation limit 2;
> +--+
> | N_REGIONKEY  |
> +--+
> | null |
> | null |
> +--+
> 2 rows selected (0.25 seconds)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4894) Fix unit test failure in 'storage-hive/core' module

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4894.

   Resolution: Fixed
Fix Version/s: 1.9.0

Fixed in 
[f3c26e3|https://github.com/apache/drill/commit/f3c26e34e3a72ef338c4dbca1a0204f342176972]

> Fix unit test failure in 'storage-hive/core' module
> ---
>
> Key: DRILL-4894
> URL: https://issues.apache.org/jira/browse/DRILL-4894
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
> Fix For: 1.9.0
>
>
> As part of DRILL-4886, I added `hbase-server` as a dependency for 
> 'storage-hive/core' which pulled older version (2.5.1) of some hadoop jars, 
> incompatible with other hadoop jars used by drill (2.7.1).
> This breaks unit tests in this module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-3178) csv reader should allow newlines inside quotes

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-3178.

   Resolution: Fixed
Fix Version/s: (was: Future)
   1.9.0

Fixed in 
[42948fe|https://github.com/apache/drill/commit/42948feb4a45f98f3d116d2e2a765cc3fadb5937]

> csv reader should allow newlines inside quotes 
> ---
>
> Key: DRILL-3178
> URL: https://issues.apache.org/jira/browse/DRILL-3178
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Text & CSV
>Affects Versions: 1.0.0
> Environment: Ubuntu Trusty 14.04.2 LTS
>Reporter: Neal McBurnett
>Assignee: F Méthot
> Fix For: 1.9.0
>
> Attachments: drill-3178.patch
>
>
> When reading a csv file which contains newlines within quoted strings, e.g. 
> via
> select * from dfs.`/tmp/q.csv`;
> Drill 1.0 says:
> Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException:  
> Error processing input: Cannot use newline character within quoted string
> But many tools produce csv files with newlines in quoted strings.  Drill 
> should be able to handle them.
> Workaround: the csvquote program (https://github.com/dbro/csvquote) can 
> encode embedded commas and newlines, and even decode them later if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4653.

   Resolution: Fixed
Fix Version/s: (was: Future)
   1.9.0

Fixed in 
[db48298|https://github.com/apache/drill/commit/db48298920575cb1c2283e03bdfc7b50e83ae217]

> Malformed JSON should not stop the entire query from progressing
> 
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.6.0
>Reporter: subbu srinivasan
> Fix For: 1.9.0
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4369) Database driver fails to report any major or minor version information

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4369.

Resolution: Fixed

> Database driver fails to report any major or minor version information
> --
>
> Key: DRILL-4369
> URL: https://issues.apache.org/jira/browse/DRILL-4369
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.4.0
>Reporter: N Campbell
>Assignee: Laurent Goujon
> Fix For: 1.9.0
>
>
> Using Apache 1.4 Drill
> The DatabaseMetadata.getters to obtain the Major and Minor versions of the 
> server or JDBC driver return 0 instead of 1.4.
> This prevents an application from dynamically adjusting how it interacts 
> based on which version of Drill a connection is accessing.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

The mailing list does not seem to allow for images. Can you put the image 
elsewhere (Github or Dropbox), and reply with a link to it?

- Sudheesh

> On Oct 19, 2016, at 5:37 PM, François Méthot <fmetho...@gmail.com> wrote:
> 
> We had problem on the 220 nodes cluster. No problem on the 12 nodes cluster.
> 
> I agree that the data may not be distributed evenly. It would be a long and 
> tedious process for me to produce a report. 
> 
> Here is a drawing  of the fragments overview before and after the changes of 
> the affinity factory on a sample query ran on the 220 nodes cluster.  
> max_width_per_node=8 on both, but it turned out to be irrelevant to the issue.
> 
> 
> 
> 
> 
> 
> Before: SYSTEM ERROR: ForemanException: One more more nodes lost connectivity 
> during query.  Identified nodes were [server121:31010].
> 
> After: error is gone
> 
> Before: low disk io, high network io on the bottom part of the graph
> after : high disk io, low network io on the bottom part of the graph
> 
> 
> 
> 
> 
> 
> 
> On Tue, Oct 18, 2016 at 12:58 AM, Padma Penumarthy <ppenumar...@maprtech.com 
> <mailto:ppenumar...@maprtech.com>> wrote:
> Hi Francois,
> 
> It would be good to understand how increasing affinity_factor helped in your 
> case
> so we can better document and also use that knowledge to improve things in 
> future release.
> 
> If you have two clusters,  it is not clear whether you had the problem on 12 
> node cluster
> or 220 node cluster or both. Is the dataset same on both ? Is 
> max_width_per_node=8 in both clusters ?
> 
> Increasing affinity factor will lower remote reads  by scheduling more 
> fragments/doing more work
> on nodes which have data available locally.  So, there seem to be some kind 
> of non uniform
> data distribution for sure. It would be good if you can provide more details 
> i.e. how the data is
> distributed in the cluster and how the load on the nodes changed when 
> affinity factor was increased.
> 
> Thanks,
> Padma
> 
> 
> > On Oct 14, 2016, at 6:45 PM, François Méthot <fmetho...@gmail.com 
> > <mailto:fmetho...@gmail.com>> wrote:
> >
> > We have  a 12 nodes cluster and a 220 nodes cluster, but they do not talk
> > to each other. So Padma's analysis do not apply but thanks for your
> > comments. Our goal had been to run Drill on the 220 nodes cluster after it
> > proved worthy of it on the small cluster.
> >
> > planner.width.max_per_node was eventually reduced to 2 when we were trying
> > to figure this out, it would still fail. After we figured out the
> > affinity_factor, we put it back to its original value and it would work
> > fine.
> >
> >
> >
> > Sudheesh: Indeed, The Zk/drill services use the same network on our bigger
> > cluster.
> >
> > potential improvements:
> > - planner.affinity_factor should be better documented.
> > - When ZK disconnected, the running queries systematically failed. When we
> > disabled the ForemanException thrown in the QueryManager.
> > drillbitUnregistered method, most of our query started to run successfully,
> > we would sometime get Drillbit Disconnected error within the rpc work bus.
> > It did confirm that we still had something on our network going on, but it
> > also showed that the RPC bus between drillbits was more resilient to
> > network hiccup. I could not prove it, but I think under certain condition,
> > the ZK session gets recreated, which cause a Query Manager unregistered
> > (query fail) and register call right after, but the RPC
> > bus  would remains connected.
> >
> >
> > We really appreciate your feedback and we hope to contribute to this great
> > project in the future.
> > Thanks
> > Francois
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 14, 2016 at 3:00 PM, Padma Penumarthy <ppenumar...@maprtech.com 
> > <mailto:ppenumar...@maprtech.com>>
> > wrote:
> >
> >>
> >> Seems like you have 215 nodes, but the data for your query is there on
> >> only 12 nodes.
> >> Drill tries to distribute the scan fragments across the cluster more
> >> uniformly (trying to utilize all CPU resources).
> >> That is why you have lot of remote reads going on and increasing affinity
> >> factor eliminates running scan
> >> fragments on the other (215-12) nodes.
> >>
> >> you also mentioned planner.width.max_per_node is set to 8.
> >> So, with increased affinity factor,  you have 8 scan fragments doing a lot
> >> more work on these 12 node

[jira] [Created] (DRILL-4950) Consume Spurious Empty Batches in JDBC

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4950:
--

 Summary: Consume Spurious Empty Batches in JDBC
 Key: DRILL-4950
 URL: https://issues.apache.org/jira/browse/DRILL-4950
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - JDBC
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Priority: Blocker
 Fix For: 1.9.0


In 
[DrillCursor|https://github.com/apache/drill/blob/master/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L199],
 consume all empty batches, not just non-continuous empty batches. This results 
in query cancellation (from sqlline) and incomplete results.

Introduced (regression?) in DRILL-2548.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

 server064:31010
>>> 
>>> <--- 30 seconds gap for that fragment --->
>>> 
>>> 2016-09-26 20:37:09,976 [BitServer-2] WARN 
>>> o.a.d.exec.rpc.control.ControlServer
>>> - Message of mode REQUEST of rpc type 2 took longer then 500 ms. Actual
>>> duration was 23617ms.
>>> 
>>> 2016-09-26 20:07:38,211 [...uuid...frag:1:10] DEBUG
>>> o.a.d.e.p.i.s.RemovingRecordBatch - doWork(): 0 records copied out of 0,
>>> remaining: 0 incoming schema BatchSchema [, selectionVector=TWO_BYTE]
>>> 2016-09-26 20:07:38,211 [...uuid...frag:1:10] DEBUG
>>> o.a.d.exec.rpc.control.WorkEventBus - Cancelling and removing fragment
>>> manager : ...uuid...
>>> 
>>> 
>>> 
>>> For the same query on a working node:
>>> 2016-09-26 20:07:09,056 [...uuid...frag:1:2] INFO
>>> o.a.d.e.w.f.FragmentStatusReporter - ...uuid...:1:2: State to report:
>>> RUNNING
>>> 2016-09-26 20:07:09,056 [...uuid...frag:1:2] DEBUG
>>> o.a.d.e.w.FragmentExecutor - Starting fragment 1:2 on server125:31010
>>> 2016-09-26 20:07:09,749 [...uuid...frag:1:2] DEBUG
>>> o.a.d.e.p.i.s.RemovingRecordBatch - doWork(): 0 records copied out of 0,
>>> remaining: 0 incoming schema BatchSchema [, selectionVector=TWO_BYTE]
>>> 2016-09-26 20:07:09,749 [...uuid...frag:1:2] DEBUG
>>> o.a.d.e.p.i.s.RemovingRecordBatch - doWork(): 0 records copied out of 0,
>>> remaining: 0 incoming schema BatchSchema [, selectionVector=TWO_BYTE]
>>> 2016-09-26 20:07:11,005 [...uuid...frag:1:2] DEBUG
>>> o.a.d.e.s.p.c.ParquetRecordReader - Read 87573 records out of row
>>> groups(0) in file `/data/drill/tmp/PARQUET_EMPLOYEE/0_0_14.parquet
>>> 
>>> 
>>> 
>>> 
>>> We are investigating what could get cause that 30 seconds gap for that
>>> fragment.
>>> 
>>> Any idea let us know
>>> 
>>> Thanks
>>> Francois
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Sep 19, 2016 at 2:59 PM, François Méthot <fmetho...@gmail.com>
>>> wrote:
>>> 
>>>> Hi Sudheesh,
>>>> 
>>>>  If I add selection filter so that no row are returned, the same
>>>> problem occur. I also simplified the query to include only few integer
>>>> columns.
>>>> 
>>>> That particular data repo is ~200+ Billions records spread over ~50 000
>>>> parquet files.
>>>> 
>>>> We have other CSV data repo that are 100x smaller that does not trigger
>>>> this issue.
>>>> 
>>>> 
>>>> + Is atsqa4-133.qa.lab [1] the Foreman node for the query in this case?
>>>> There is also a bizarre case where the node that is reported as lost is the
>>>> node itself.
>>>> Yes, the stack trace is from the ticket, It did occurred once or twice
>>>> (in the many many attempts) that it was the node itself.
>>>> 
>>>> + Is there a spike in memory usage of the Drillbit this is the Foreman
>>>> for the query (process memory, not just heap)?
>>>> We don't notice any unusual spike, each nodes gets busy in the same
>>>> range when query is running.
>>>> 
>>>> I tried running with 8GB/20GB and 4GB/24GB heap/off-heap, did not see
>>>> any improvement.
>>>> 
>>>> 
>>>> We will update from 1.7 to 1.8 before going ahead with more
>>>> investigation.
>>>> 
>>>> Thanks a lot.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Mon, Sep 19, 2016 at 1:19 PM, Sudheesh Katkam <skat...@maprtech.com>
>>>> wrote:
>>>> 
>>>>> Hi Francois,
>>>>> 
>>>>> A simple query with only projections is not an “ideal” use case, since
>>>>> Drill is bound by how fast the client can consume records. There are 1000
>>>>> scanners sending data to 1 client (vs far fewer scanners sending data in
>>>>> the 12 node case).
>>>>> 
>>>>> This might increase the load on the Drillbit that is the Foreman for
>>>>> the query. In the query profile, the scanners should 

Hangout starting now..

[Sudheesh Katkam]

Link: https://hangouts.google.com/hangouts/_/maprtech.com/drillbi-weeklyhangout 


Thank you,
Sudheesh

Re: [HANGOUT] Topics for 10/04/16

[Sudheesh Katkam]

Join us at this link:

https://hangouts.google.com/hangouts/_/maprtech.com/drillbi-weeklyhangout 
<https://hangouts.google.com/hangouts/_/maprtech.com/drillbi-weeklyhangout>

> On Oct 3, 2016, at 9:26 PM, Shadi Khalifa <khal...@cs.queensu.ca> wrote:
> 
> Hi,
> I have been working on integrating WEKA into Drill to support building and 
> scoring classification models. I have been successful in supporting all WEKA 
> classifiers and making them run in a distributed fashion over Drill 1.2. The 
> classifier accuracy is not affected by running in a distributed fashion and 
> the training and scoring times are getting a huge boost using Drill. A paper 
> on this has been published in the IEEE symposium on Big Data in June 2016 
> [available: 
> http://cs.queensu.ca/~khalifa/qdrill/QDrill_20160212IEEE_CameraReady.pdf] and 
> we are now in the process of publishing another paper in which QDrill 
> supports all WEKA algorithms. FYI, this can be easily extended to support 
> clustering and other types of WEKA algorithms. The architecture also allows 
> supporting other data mining libraries.
> The QDrill project website is  http://cs.queensu.ca/~khalifa/qdrill, the 
> project downloadable version on it is little bit old but I'm planning to 
> upload a more updated stable version within the next 10 days. I'm also using 
> an SVN repository and planning to move the project to GitHub to make it 
> easier to get the latest Drill versions and to may be integrate with Drill at 
> some point. 
> Unfortunately, I have another meeting tomorrow at the same time of the 
> hangout, but I would love to know your opinion and to discuss the process of 
> evaluating this extension and may be integrating it with Drill at some point. 
> Regards
> Shadi KhalifaPhD CandidateSchool of Computing Queen's University Canada
> I'm just a neuron in the society collective brain
> 
> 01001001 0010 01101100 0110 01110110 01100101 0010 01000101 
> 01100111 0001 0111 01110100 
> P Please consider your environmental responsibility before printing this 
> e-mail
> 
> 
> 
>On Monday, October 3, 2016 10:52 PM, Laurent Goujon <laur...@dremio.com> 
> wrote:
> 
> 
> Hi,
> 
> I'm currently working on improving metadata support for both the JDBC
> driver and the C++ connector, more specifically the following JIRAs:
> 
> DRILL-4853: Update C++ protobuf source files
> DRILL-4420: Server-side metadata and prepared-statement support for C++
> connector
> DRILL-4880: Support JDBC driver registration using ServiceLoader
> DRILL-4925: Add tableType filter to GetTables metadata query
> DRILL-4730: Update JDBC DatabaseMetaData implementation to use new Metadata
> APIs
> 
> I  already opened multiple pull requests for those (the list is available
> at https://github.com/apache/drill/pulls/laurentgo)
> 
> I'm planning to join tomorrow hangout in case people have questions about
> those.
> 
> Cheers,
> 
> Laurent
> 
> On Mon, Oct 3, 2016 at 10:28 AM, Subbu Srinivasan <ssriniva...@zscaler.com>
> wrote:
> 
>> Can we close on https://github.com/apache/drill/pull/518 ?
>> 
>> On Mon, Oct 3, 2016 at 10:27 AM, Sudheesh Katkam <sudhe...@apache.org>
>> wrote:
>> 
>>> Hi drillers,
>>> 
>>> Our bi-weekly hangout is tomorrow (10/04/16, 10 AM PT). If you have any
>>> suggestions for hangout topics, you can add them to this thread. We will
>>> also ask around at the beginning of the hangout for topics.
>>> 
>>> Thank you,
>>> Sudheesh
>>> 
>> 
> 
> 

[HANGOUT] Topics for 10/04/16

[Sudheesh Katkam]

Hi drillers,

Our bi-weekly hangout is tomorrow (10/04/16, 10 AM PT). If you have any
suggestions for hangout topics, you can add them to this thread. We will
also ask around at the beginning of the hangout for topics.

Thank you,
Sudheesh

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

One more interesting thing and another guess to resolve the problem,

> P.S.:
> We do see this also:
> 2016-09-19 14:48:23,444 [drill-executor-9] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment ..:1:2 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-11] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:222 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-12] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:442 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-10] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:662 not found in the
> work bus.

+ Are there 220 nodes in your cluster?

If that is the case and these fragments are scanners, they seem be be assigned 
in a round robin fashion (and not based on data locality?). If so, please 
upgrade to the latest version where DRILL-4446 [1] is resolved.

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-4446 
<https://issues.apache.org/jira/browse/DRILL-4446>

> On Sep 19, 2016, at 10:19 AM, Sudheesh Katkam <skat...@maprtech.com> wrote:
> 
> Hi Francois,
> 
> A simple query with only projections is not an “ideal” use case, since Drill 
> is bound by how fast the client can consume records. There are 1000 scanners 
> sending data to 1 client (vs far fewer scanners sending data in the 12 node 
> case).
> 
> This might increase the load on the Drillbit that is the Foreman for the 
> query. In the query profile, the scanners should be spending a lot more time 
> “waiting” to send records to the client (via root fragment).
> + Is atsqa4-133.qa.lab [1] the Foreman node for the query in this case? There 
> is also a bizarre case where the node that is reported as lost is the node 
> itself.
> + Is there a spike in memory usage of the Drillbit this is the Foreman for 
> the query (process memory, not just heap)?
> 
> Regarding the warnings ...
> 
>> 2016-09-19 13:31:56,866 [BitServer-7] WARN
>> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type
>> 6 took longer than 500 ms. Actual Duration was 16053ms.
> 
> 
> RPC type 6 is a cancellation request; DRILL-4766 [2] should help in this 
> case, which is resolved in the latest version of Drill. So as Chun suggested, 
> upgrade the cluster to the latest version of Drill.
> 
>> 2016-09-19 14:15:33,357 [BitServer-4] WARN
>> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type
>> 1 took longer than 500 ms. Actual Duration was 981ms.
> 
> I am surprised that responses are taking that long to handle.
> + Are both messages on the same Drillbit?
> 
> The other warnings can be ignored.
> 
> Thank you,
> Sudheesh
> 
> [1] I just realized that atsqa4-133.qa.lab is in one of our test environments 
> :)
> [2] https://issues.apache.org/jira/browse/DRILL-4766 
> <https://issues.apache.org/jira/browse/DRILL-4766>
> 
>> On Sep 19, 2016, at 9:15 AM, François Méthot <fmetho...@gmail.com 
>> <mailto:fmetho...@gmail.com>> wrote:
>> 
>> Hi Sudheesh,
>> 
>> 
>> + Does the query involve any aggregations or filters? Or is this a select
>> query with only projections?
>> Simple query with only projections
>> 
>> + Any suspicious timings in the query profile?
>> Nothing specially different than our working query on our small cluster.
>> 
>> + Any suspicious warning messages in the logs around the time of failure on
>> any of the drillbits? Specially on atsqa4-133.qa.lab? Specially this one
>> (“..” are place holders):
>>  Message of mode .. of rpc type .. took longer than ..ms.  Actual duration
>> was ..ms.
>> 
>> Well we do see this warning on the failing node (on my last test), I found
>> this WARNING in our log for the past month for pretty much every node I
>> checked.
>> 2016-09-19 13:31:56,866 [BitServer-7] WARN
>> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type
>> 6 took longer than 500 ms. Actual Duration was 16053ms.
>> 2016-09-19 14:15:33,357 [BitServer-4] WARN
>> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type
>> 1 took longer than 500 ms. Actual Duration was 981ms.
>> 
>> We really appreciate your help. I will dig in the source code for when and
>> why this error happen.
>> 
>> 
>> Francois
>> 
>> P.S.:
>> We do see this also:
>> 2016-09-19 14:48:23,444 [drill-executor-9] WARN
>> o.a.d.exec.rpc.control.WorkEventBus - Fragment ..:1:2 not found in the
>> work bus.
>> 2016-09-19 14:48:23,444 [drill-executor-11] WARN
>> o.a.d.exec.rpc.c

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

Hi Francois,

A simple query with only projections is not an “ideal” use case, since Drill is 
bound by how fast the client can consume records. There are 1000 scanners 
sending data to 1 client (vs far fewer scanners sending data in the 12 node 
case).

This might increase the load on the Drillbit that is the Foreman for the query. 
In the query profile, the scanners should be spending a lot more time “waiting” 
to send records to the client (via root fragment).
+ Is atsqa4-133.qa.lab [1] the Foreman node for the query in this case? There 
is also a bizarre case where the node that is reported as lost is the node 
itself.
+ Is there a spike in memory usage of the Drillbit this is the Foreman for the 
query (process memory, not just heap)?

Regarding the warnings ...

> 2016-09-19 13:31:56,866 [BitServer-7] WARN
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type
> 6 took longer than 500 ms. Actual Duration was 16053ms.


RPC type 6 is a cancellation request; DRILL-4766 [2] should help in this case, 
which is resolved in the latest version of Drill. So as Chun suggested, upgrade 
the cluster to the latest version of Drill.

> 2016-09-19 14:15:33,357 [BitServer-4] WARN
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type
> 1 took longer than 500 ms. Actual Duration was 981ms.

I am surprised that responses are taking that long to handle.
+ Are both messages on the same Drillbit?

The other warnings can be ignored.

Thank you,
Sudheesh

[1] I just realized that atsqa4-133.qa.lab is in one of our test environments :)
[2] https://issues.apache.org/jira/browse/DRILL-4766 
<https://issues.apache.org/jira/browse/DRILL-4766>

> On Sep 19, 2016, at 9:15 AM, François Méthot <fmetho...@gmail.com> wrote:
> 
> Hi Sudheesh,
> 
> 
> + Does the query involve any aggregations or filters? Or is this a select
> query with only projections?
> Simple query with only projections
> 
> + Any suspicious timings in the query profile?
> Nothing specially different than our working query on our small cluster.
> 
> + Any suspicious warning messages in the logs around the time of failure on
> any of the drillbits? Specially on atsqa4-133.qa.lab? Specially this one
> (“..” are place holders):
>  Message of mode .. of rpc type .. took longer than ..ms.  Actual duration
> was ..ms.
> 
> Well we do see this warning on the failing node (on my last test), I found
> this WARNING in our log for the past month for pretty much every node I
> checked.
> 2016-09-19 13:31:56,866 [BitServer-7] WARN
> o.a.d.exec.rpc.control.ControlServer - Message of mode REQUEST of rpc type
> 6 took longer than 500 ms. Actual Duration was 16053ms.
> 2016-09-19 14:15:33,357 [BitServer-4] WARN
> o.a.d.exec.rpc.control.ControlClient - Message of mode RESPONSE of rpc type
> 1 took longer than 500 ms. Actual Duration was 981ms.
> 
> We really appreciate your help. I will dig in the source code for when and
> why this error happen.
> 
> 
> Francois
> 
> P.S.:
> We do see this also:
> 2016-09-19 14:48:23,444 [drill-executor-9] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment ..:1:2 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-11] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:222 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-12] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:442 not found in the
> work bus.
> 2016-09-19 14:48:23,444 [drill-executor-10] WARN
> o.a.d.exec.rpc.control.WorkEventBus - Fragment :1:662 not found in the
> work bus.
> 
> 
> 
> 
> On Sun, Sep 18, 2016 at 2:57 PM, Sudheesh Katkam <skat...@maprtech.com>
> wrote:
> 
>> Hi Francois,
>> 
>> More questions..
>> 
>>> + Can you share the query profile?
>>>  I will sum it up:
>>> It is a select on 18 columns: 9 string, 9 integers.
>>> Scan is done on 13862 parquet files spread  on 1000 fragments.
>>> Fragments are spread accross 215 nodes.
>> 
>> So ~5 leaf fragments (or scanners) per Drillbit seems fine.
>> 
>> + Does the query involve any aggregations or filters? Or is this a select
>> query with only projections?
>> + Any suspicious timings in the query profile?
>> + Any suspicious warning messages in the logs around the time of failure
>> on any of the drillbits? Specially on atsqa4-133.qa.lab? Specially this one
>> (“..” are place holders):
>>  Message of mode .. of rpc type .. took longer than ..ms.  Actual
>> duration was ..ms.
>> 
>> Thank you,
>> Sudheesh
>> 
>>> On Sep 15, 2016, at 11:27 AM, François Méthot <fmetho...@gmail.com>
>> wrote:
>>> 
>>> Hi Sudheesh,
>

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

Hi Francois,

More questions..

> + Can you share the query profile?
>   I will sum it up:
>  It is a select on 18 columns: 9 string, 9 integers.
>  Scan is done on 13862 parquet files spread  on 1000 fragments.
>  Fragments are spread accross 215 nodes.

So ~5 leaf fragments (or scanners) per Drillbit seems fine.

+ Does the query involve any aggregations or filters? Or is this a select query 
with only projections?
+ Any suspicious timings in the query profile?
+ Any suspicious warning messages in the logs around the time of failure on any 
of the drillbits? Specially on atsqa4-133.qa.lab? Specially this one (“..” are 
place holders):
  Message of mode .. of rpc type .. took longer than ..ms.  Actual duration was 
..ms.

Thank you,
Sudheesh

> On Sep 15, 2016, at 11:27 AM, François Méthot <fmetho...@gmail.com> wrote:
> 
> Hi Sudheesh,
> 
> + How many zookeeper servers in the quorum?
> The quorum has 3 servers, everything looks healthy
> 
> + What is the load on atsqa4-133.qa.lab when this happens? Any other
> applications running on that node? How many threads is the Drill process
> using?
> The load on the failing node(8 cores) is 14, when Drill is running. Which
> is nothing out of the ordinary according to our admin.
> HBase is also running.
> planner.width.max_per_node is set to 8
> 
> + When running the same query on 12 nodes, is the data size same?
> Yes
> 
> + Can you share the query profile?
>   I will sum it up:
>  It is a select on 18 columns: 9 string, 9 integers.
>  Scan is done on 13862 parquet files spread  on 1000 fragments.
>  Fragments are spread accross 215 nodes.
> 
> 
> We are in process of increasing our Zookeeper session timeout config to see
> if it helps.
> 
> thanks
> 
> Francois
> 
> 
> 
> 
> 
> 
> 
> On Wed, Sep 14, 2016 at 4:40 PM, Sudheesh Katkam <skat...@maprtech.com>
> wrote:
> 
>> Hi Francois,
>> 
>> Few questions:
>> + How many zookeeper servers in the quorum?
>> + What is the load on atsqa4-133.qa.lab when this happens? Any other
>> applications running on that node? How many threads is the Drill process
>> using?
>> + When running the same query on 12 nodes, is the data size same?
>> + Can you share the query profile?
>> 
>> This may not be the right thing to do, but for now, If the cluster is
>> heavily loaded, increase the zk timeout.
>> 
>> Thank you,
>> Sudheesh
>> 
>>> On Sep 14, 2016, at 11:53 AM, François Méthot <fmetho...@gmail.com>
>> wrote:
>>> 
>>> We are running 1.7.
>>> The log were taken from the jira tickets.
>>> 
>>> We will try out 1.8 soon.
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Sep 14, 2016 at 2:52 PM, Chun Chang <cch...@maprtech.com> wrote:
>>> 
>>>> Looks like you are running 1.5. I believe there are some work done in
>> that
>>>> area and the newer release should behave better.
>>>> 
>>>> On Wed, Sep 14, 2016 at 11:43 AM, François Méthot <fmetho...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> We are trying to find a solution/workaround to issue:
>>>>> 
>>>>> 2016-01-28 16:36:14,367 [Curator-ServiceCache-0] ERROR
>>>>> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: ForemanException:
>>>>> One more more nodes lost connectivity during query.  Identified nodes
>>>>> were [atsqa4-133.qa.lab:31010].
>>>>> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
>>>>> ForemanException: One more more nodes lost connectivity during query.
>>>>> Identified nodes were [atsqa4-133.qa.lab:31010].
>>>>>   at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
>>>>> close(Foreman.java:746)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>   at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> processEvent(Foreman.java:858)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>   at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> processEvent(Foreman.java:790)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>   at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>>>> moveToState(Foreman.java:792)
>>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>>>   at org.apache.drill.exec.work.foreman.Foreman.moveToState(
>>>>> Foreman.java:9

Re: System/session options

[Sudheesh Katkam]

Can you provide more details about your case?

DRILL-3363 requests for a nice error message for options that cannot be set at 
session level (there is no handle to a UserSession in some cases e.g. function 
registry). AFAIK currently, such statements are no ops.

Thank you,
Sudheesh

> On Sep 16, 2016, at 8:55 AM, Vitalii Diravka  
> wrote:
> 
> Hi all!
> 
> I am going to add one new option and it looks like I can use it only at the
> system level (Metadata class).
> 
> I saw this task https://issues.apache.org/jira/browse/DRILL-3363.
> Does it mean that only system-wide-variables could be used in drill
> (without appropriate session options)?
> 
> 
> Kind regards
> Vitalii

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

Hi Francois,

Few questions:
+ How many zookeeper servers in the quorum?
+ What is the load on atsqa4-133.qa.lab when this happens? Any other 
applications running on that node? How many threads is the Drill process using?
+ When running the same query on 12 nodes, is the data size same?
+ Can you share the query profile?

This may not be the right thing to do, but for now, If the cluster is heavily 
loaded, increase the zk timeout.

Thank you,
Sudheesh

> On Sep 14, 2016, at 11:53 AM, François Méthot  wrote:
> 
> We are running 1.7.
> The log were taken from the jira tickets.
> 
> We will try out 1.8 soon.
> 
> 
> 
> 
> On Wed, Sep 14, 2016 at 2:52 PM, Chun Chang  wrote:
> 
>> Looks like you are running 1.5. I believe there are some work done in that
>> area and the newer release should behave better.
>> 
>> On Wed, Sep 14, 2016 at 11:43 AM, François Méthot 
>> wrote:
>> 
>>> Hi,
>>> 
>>>  We are trying to find a solution/workaround to issue:
>>> 
>>> 2016-01-28 16:36:14,367 [Curator-ServiceCache-0] ERROR
>>> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: ForemanException:
>>> One more more nodes lost connectivity during query.  Identified nodes
>>> were [atsqa4-133.qa.lab:31010].
>>> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
>>> ForemanException: One more more nodes lost connectivity during query.
>>> Identified nodes were [atsqa4-133.qa.lab:31010].
>>>at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
>>> close(Foreman.java:746)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> processEvent(Foreman.java:858)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> processEvent(Foreman.java:790)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> moveToState(Foreman.java:792)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman.moveToState(
>>> Foreman.java:909)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman.access$2700(
>>> Foreman.java:110)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateListener.
>>> moveToState(Foreman.java:1183)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>> 
>>> 
>>> DRILL-4325  
>>> ForemanException:
>>> One or more nodes lost connectivity during query
>>> 
>>> 
>>> 
>>> Any one experienced this issue ?
>>> 
>>> It happens when running query involving many parquet files on a cluster
>> of
>>> 200 nodes. Same query on a smaller cluster of 12 nodes runs fine.
>>> 
>>> It is not caused by garbage collection, (checked on both ZK node and the
>>> involved drill bit).
>>> 
>>> Negotiated max session timeout is 40 seconds.
>>> 
>>> The sequence seems:
>>> - Drill Query begins, using an existing ZK session.
>>> - Drill Zk session timeouts
>>>  - perhaps it was writing something that took too long
>>> - Drill attempts to renew session
>>>   - drill believes that the write operation failed, so it attempts
>> to
>>> re-create the zk node, which trigger another exception.
>>> 
>>> We are open to any suggestion. We will report any finding.
>>> 
>>> Thanks
>>> Francois
>>> 
>> 

Re: DRILL JDBC Driver setFetchSize

[Sudheesh Katkam]

Hi Sudip,

fetchSize is number of records to fetch per network call from the server (to 
populate a ResultSet), not the same as LIMIT clause. See [1].

Currently, the client does not advertise this to the server, and the server 
does not have this capability.

Thank you,
Sudheesh

[1] 
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setFetchSize(int)
 


> On Sep 13, 2016, at 11:48 PM, Sudip Mukherjee  
> wrote:
> 
> Hi,
> 
> Should the setFetchSize work similar to applying LIMIT clause with drill JDBC 
> driver? I've set setFetchSize to 
> java.sql.Statement
>  but the query still gets records beyond the fetchSIze.
> 
> Thanks,
> Sudip

Re: Nested/n-dimensional Arrays in Mongo/Json and Drill

[Sudheesh Katkam]

Hi Pradeeban,

Can you post the detailed error message?

First set the option:
> SET `exec.errors.verbose` = true;

And then run the query. The detailed output will point us to where the error 
occurred.

Thank you,
Sudheesh

> On Sep 12, 2016, at 10:47 AM, Pradeeban Kathiravelu  
> wrote:
> 
> Hi,
> I have a complex json data stored in Mongo.
> 
> The data has nested arrays as shown below .
> 
> here when I try to access the geometry field in select queries, the Drill
> query fails.
> 
> select camic._id, camic.type, camic.parent_id, camic.randval,
> camic.creation_date, camic.object_type, camic.x, camic.y, camic.normalized,
> camic.bbox, camic.geometry, camic.footprint, camic.properties,
> camic.provenance, camic.submit_date from mongo.CAMICROSCOPE.`testUAIM2` as
> camic WHERE ((camic.provenance.image.case_id = 'TCGA-02-0001-01Z-00-DX1')
> AND (camic.provenance.analysis.execution_id  = 'tammy-test:7') AND
> (camic.footprint >= 800) AND (camic.x >= 0) AND (camic.x <=1) AND (camic.y
>> = 0) AND (camic.y <= 1));
> 
> would fail with a below error:
> 
> 
> 
> 
> 
> *Error: SYSTEM ERROR: NullPointerExceptionFragment 0:0[Error Id:
> 8cd950af-91fa-4cf8-865b-265f227c8e87 on llovizna:31010] (state=,code=0)*
> 
> However, after removing the geometry from the select query, it would work
> just fine.
> 
> select camic._id, camic.type, camic.parent_id, camic.randval,
> camic.creation_date, camic.object_type, camic.x, camic.y, camic.normalized,
> camic.bbox, camic.footprint, camic.properties, camic.provenance,
> camic.submit_date from mongo.CAMICROSCOPE.`testUAIM2` as camic WHERE
> ((camic.provenance.image.case_id = 'TCGA-02-0001-01Z-00-DX1') AND
> (camic.provenance.analysis.execution_id  = 'tammy-test:7') AND
> (camic.footprint >= 800) AND (camic.x >= 0) AND (camic.x <=1) AND (camic.y
>> = 0) AND (camic.y <= 1));
> 
> 
> I know this is due to the array in the output. I am also aware of the
> commonly suggested options.
> 1. Using the array indexes in the select query: This is *impractical*. I do
> not know how many elements I would have in this geojson - the coordinates.
> It may be millions or as low as 3.
> 
> 2. Flatten keyword: I am using Drill on top of Mongo - and finding an
> interesting case where Drill outperforms certain queries in a distributed
> execution than just using Mongo. Using Flatten basically kills all the
> performance benefits I have with Drill otherwise. Flatten is just plain
> *expensive* operation for the scale of my data (around 48 GB. But I can
> split them into a few GB each).
> 
> Now given that I have explained why I cannot use the commonly suggested
> options to deal with the complex Drill queries involving arrays, what are
> the other alternatives?
> 
> I am thinking of using Protobufs with Drill to serialize the nested arrays.
> Has anyone else tried it before, and any pointers, or anyother suggestions
> on this overall requirement of querying the nested arrays?
> 
> Thank you.
> Regards,
> Pradeeban.
> 
> 
>  The sample data in Mongo (data anonymized).
>> db.testUAIM2.findOne()
> {
>"_id" : ObjectId("22"),
>"bbox" : [
>0.11,
>0.33,
>0.44,
>0.44
>],
>"*geometry*" : {
>"type" : "Polygon",
>"coordinates" : [
>[
>[
>0.04272915795445442,
>0.9368849396705627
>],
>[
>0.04272915795445442,
>0.9369083046913147
>],
>[
>0.042739588767290115,
>0.9369083046913147
>],
>[
>0.042739588767290115,
>0.9369550347328186
>],
>[
>0.04275001958012581,
>0.9369550347328186
>],
>[
>0.04275001958012581,
>0.9370251893997192
>],
>[
>0.042760446667671204,
>0.9370251893997192
>],
>[
>0.042760446667671204,
>0.9371420741081238
>],
>[
>0.04275001958012581,
>0.9371420741081238
>],
>[
>0.04275001958012581,
>0.9372121691703796
>],
>[
>0.042739588767290115,
>0.9372121691703796
>],
>[
>0.042739588767290115,
>0.9372823238372803
>],
>[
>0.04272915795445442,
>0.9372823238372803
>],
>[
>0.04272915795445442,
>0.9373056888580322
>],

[jira] [Created] (DRILL-4841) Use user server event loop group for web clients

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4841:
--

 Summary: Use user server event loop group for web clients
 Key: DRILL-4841
 URL: https://issues.apache.org/jira/browse/DRILL-4841
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - HTTP
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Priority: Minor


Currently we spawn an event loop group for handling requests from clients. This 
group should also be used to handles responses (from server) for web clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4836.

Resolution: Fixed

Fixed in 
[0a4c21c|https://github.com/apache/drill/commit/0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1].

> ZK Issue during Drillbit startup, possibly due to race condition
> 
>
> Key: DRILL-4836
> URL: https://issues.apache.org/jira/browse/DRILL-4836
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server
>Reporter: Abhishek Girish
>Assignee: Paul Rogers
> Fix For: 1.8.0
>
>
> During a parallel launch of Drillbits on a 4 node cluster, I hit this issue 
> during startup:
> {code}
> Exception in thread "main" 
> org.apache.drill.exec.exception.DrillbitStartupException: Failure during 
> initial startup of Drillbit.
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257)
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable 
> to put
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196)
> at 
> org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94)
> ...
> at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281)
> ... 2 more
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs
> at 
> org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191)
> ... 7 more
> {code}
> And similarly,
> {code}
> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: 
> KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4822) Extend distrib-env.sh search to consider site directory

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4822.

Resolution: Fixed

Fixed in 
[4bd67a6|https://github.com/apache/drill/commit/4bd67a66073a429fb19c4f21f4113fef8a24db21].

> Extend distrib-env.sh search to consider site directory
> ---
>
> Key: DRILL-4822
> URL: https://issues.apache.org/jira/browse/DRILL-4822
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Priority: Minor
>
> DRILL-4581 provided revisions to the Drill launch scripts. As part of that 
> fix, we introduced a new distrib-env.sh file to hold settings created by 
> custom Drill installers (that is, by custom distributions.) The original 
> version of this feature looks for distrib-env.sh only in $DRILL_HOME/env.
> Experience suggests that installers will write site-specific values to 
> distrib-env.sh and so the file must then be copied to $DRILL_SITE when 
> running under YARN. Add $DRILL_SITE to the search path in drill-config.sh for 
> distrib-env.sh.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4623) Disable Epoll by Default

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4623.

Resolution: Fixed

Fixed in 
[48b3fd2|https://github.com/apache/drill/commit/48b3fd2ee040f30be3ff3172bf03111ca58d4b48].

> Disable Epoll by Default
> 
>
> Key: DRILL-4623
> URL: https://issues.apache.org/jira/browse/DRILL-4623
> Project: Apache Drill
>  Issue Type: Bug
>    Reporter: Sudheesh Katkam
>        Assignee: Sudheesh Katkam
>
> At higher concurrency (and/or spuriously), we hit [netty issue 
> #3539|https://github.com/netty/netty/issues/3539]. This is an issue with the 
> version of Netty Drill currently uses. Once Drill moves to a later version of 
> Netty, epoll should be reenabled by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Time for a 1.8 release

[Sudheesh Katkam]

Yes, that’s a typo. DRILL-3623 was for LIMIT 0 changes. That number will 
forever remain in my subconscious ;)

Thank you,
Sudheesh

> On Aug 5, 2016, at 1:27 PM, Zelaine Fong <zf...@maprtech.com> wrote:
> 
> I assume #3 has a typo.  It should be DRILL-4623.
> 
> -- Zelaine
> 
> On Fri, Aug 5, 2016 at 1:15 PM, Sudheesh Katkam <sudhe...@apache.org> wrote:
> 
>> Here’s my list that I will try to get in before Monday.
>> 
>> 1) DRILL-4822: Pending commit.
>> PR: https://github.com/apache/drill/pull/558
>> 
>> 2) DRILL-4819: Pending review from Aditya.
>> PR: https://github.com/apache/drill/pull/556
>> 
>> 3) DRILL-3623: Pending tests (or test results).
>> PR: https://github.com/apache/drill/pull/486
>> 
>> 4) DRILL-4792: Pending changes from Arina.
>> PR: https://github.com/apache/drill/pull/551
>> 
>> The first three will make it; the last one may not.
>> 
>> Thank you,
>> Sudheesh
>> 
>> 
>> On Wed, Aug 3, 2016 at 11:49 AM, Jinfeng Ni <jinfengn...@gmail.com> wrote:
>> 
>>> I would like to propose we set Monday 8/8 as the tentative cut-off
>>> date for 1.8 release.
>>> 
>>> If people have any working-in-progress patches, and would like to
>>> include in 1.8 release, please submit PR, or ping people to review
>>> your PR if there has been a PR under review.
>>> 
>>> Thanks,
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Aug 2, 2016 at 3:00 PM, Jinfeng Ni <jinfengn...@gmail.com>
>> wrote:
>>>> Hello everyone,
>>>> 
>>>>  I would like to start the train for 1.8 release.
>>>> 
>>>>  If you have been working on some open issues, and want to include
>>>> them in 1.8, please rely this email and let us know the JIRAs number
>>>> and corresponding pull requests.
>>>> 
>>>>  Based on the response, we may go through the list of opens JIRAs for
>>>> 1.8, and come up with a tentatively cut-off date for 1.8 release.
>>>> 
>>>>  Thanks,
>>>> 
>>>> 
>>>> 
>>>> On Tue, Aug 2, 2016 at 2:14 PM, Aditya <a...@apache.org> wrote:
>>>>> +1 for Jinfeng as the RM for 1.8 release.
>>>>> 
>>>>> On Tue, Aug 2, 2016 at 11:59 AM, Jinfeng Ni <jinfengn...@gmail.com>
>>> wrote:
>>>>> 
>>>>>> I'll volunteer to be the release manager for 1.8, if no one else
>> would
>>>>>> like to do so.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mon, Aug 1, 2016 at 9:40 PM, Parth Chandra <par...@apache.org>
>>> wrote:
>>>>>>> Hi Everyone,
>>>>>>> 
>>>>>>>   I think its time to roll out 1.8.  Would any PMC
>> member/committer
>>>>>> like
>>>>>>> to volunteer to be the release manager?
>>>>>>> 
>>>>>>> Parth
>>>>>> 
>>> 
>> 

Re: Time for a 1.8 release

[Sudheesh Katkam]

Here’s my list that I will try to get in before Monday.

1) DRILL-4822: Pending commit.
PR: https://github.com/apache/drill/pull/558

2) DRILL-4819: Pending review from Aditya.
PR: https://github.com/apache/drill/pull/556

3) DRILL-3623: Pending tests (or test results).
PR: https://github.com/apache/drill/pull/486

4) DRILL-4792: Pending changes from Arina.
PR: https://github.com/apache/drill/pull/551

The first three will make it; the last one may not.

Thank you,
Sudheesh


On Wed, Aug 3, 2016 at 11:49 AM, Jinfeng Ni  wrote:

> I would like to propose we set Monday 8/8 as the tentative cut-off
> date for 1.8 release.
>
> If people have any working-in-progress patches, and would like to
> include in 1.8 release, please submit PR, or ping people to review
> your PR if there has been a PR under review.
>
> Thanks,
>
>
>
>
> On Tue, Aug 2, 2016 at 3:00 PM, Jinfeng Ni  wrote:
> > Hello everyone,
> >
> >   I would like to start the train for 1.8 release.
> >
> >   If you have been working on some open issues, and want to include
> > them in 1.8, please rely this email and let us know the JIRAs number
> > and corresponding pull requests.
> >
> >   Based on the response, we may go through the list of opens JIRAs for
> > 1.8, and come up with a tentatively cut-off date for 1.8 release.
> >
> >   Thanks,
> >
> >
> >
> > On Tue, Aug 2, 2016 at 2:14 PM, Aditya  wrote:
> >> +1 for Jinfeng as the RM for 1.8 release.
> >>
> >> On Tue, Aug 2, 2016 at 11:59 AM, Jinfeng Ni 
> wrote:
> >>
> >>> I'll volunteer to be the release manager for 1.8, if no one else would
> >>> like to do so.
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Aug 1, 2016 at 9:40 PM, Parth Chandra 
> wrote:
> >>> > Hi Everyone,
> >>> >
> >>> >I think its time to roll out 1.8.  Would any PMC member/committer
> >>> like
> >>> > to volunteer to be the release manager?
> >>> >
> >>> > Parth
> >>>
>

[jira] [Resolved] (DRILL-4801) Setting extractHeader attribute for CSV format does not propagate to all drillbits

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4801.

Resolution: Fixed

Fixed in 
[4b362f0|https://github.com/apache/drill/commit/4b362f089637d8a6b27adc7a1b5e6c7cdb516001].

> Setting extractHeader attribute for CSV format does not propagate to all 
> drillbits 
> ---
>
> Key: DRILL-4801
> URL: https://issues.apache.org/jira/browse/DRILL-4801
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - CLI, Client - HTTP
>Affects Versions: 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
> Fix For: 1.8.0
>
>
> I have multiple drillbits running.  From web UI of one drillbit, I added 
> "extractHeader": true to the csv format.  I logged to the Web UI of a 
> different drillbit and did not see the added attributed.
> I tried the same for the TSV format and that worked as expect as the change 
> got propagated to all drillbits. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-4499) Remove unused classes

[Sudheesh Katkam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudheesh Katkam resolved DRILL-4499.

Resolution: Fixed

Fixed in 
[5a7d4c3|https://github.com/apache/drill/commit/5a7d4c3983747a778e6a29d3450dd18871e98f2c].

> Remove unused classes
> -
>
> Key: DRILL-4499
> URL: https://issues.apache.org/jira/browse/DRILL-4499
> Project: Apache Drill
>  Issue Type: Task
>    Reporter: Sudheesh Katkam
>        Assignee: Sudheesh Katkam
>
> List of unused classes that I've tracked over time:
> exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestPrune.java
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/MethodMap.java
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionBody.java
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/Multitimer.java
> exec/java-exec/src/main/java/org/apache/drill/exec/ops/QuerySetup.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/AvailabilityListener.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlCommand.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/SendProgress.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/SendProgress.java
> exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/DrillUser.java
> exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordRecorder.java
> exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/PartialWork.java
> exec/java-exec/src/main/java/org/apache/drill/exec/util/AtomicState.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/RecordOutputStream.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/ResourceRequest.java
> exec/java-exec/src/main/java/org/apache/drill/exec/work/RootNodeDriver.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Changes in Launch Scripts

[Sudheesh Katkam]

Hi all,

I just committed DRILL-4581 [1] that changes launch scripts.

The patch should be backward compatible. This email is just an FYI to start
using the new style of drill-env.sh file. The major usability change is
that Drill defaults have been moved from conf/drill-env.sh to
bin/drill-config.sh; changes to variables in drill-env.sh will override the
defaults.

See the ticket for the full list of changes.

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-4581

[DISCUSS] New Feature: Kerberos Authentication

[Sudheesh Katkam]

Hi all,

I plan to work on DRILL-4280: Kerberos Authentication for Clients [1]. The
design document [2] is attached to the ticket. Please read and comment!

Thank you,
Sudheesh

[1] https://issues.apache.org/jira/browse/DRILL-4280
[2]
https://docs.google.com/document/d/1qSBV2Hi3KwaDFADZJm9me2Nq4RqnKBsyejxunFgmdDo

[jira] [Created] (DRILL-4774) Replace NonCloseableHiveClientWithCaching

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4774:
--

 Summary: Replace NonCloseableHiveClientWithCaching
 Key: DRILL-4774
 URL: https://issues.apache.org/jira/browse/DRILL-4774
 Project: Apache Drill
  Issue Type: Bug
Reporter: Sudheesh Katkam


Likely the intention was that a NonCloseableHiveClientWithCaching object will 
be used as long as the Drill process lives.

(1) Given how StoragePluginRegistryImpl works (see createOrUpdate), 
HiveSchemaFactory leaks a NonCloseableHiveClientWithCaching every time the hive 
storage plugin is updated (because close is not invoked).

(2) I am not sure if the HiveMetastore and HiveMetaStoreClient communication 
path has a heartbeat mechanism. Otherwise when Drill is shutdown, from 
metastore point of view, at least one connection to the metastore is not closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4773) Close all HiveMetaStoreClients when HiveStoragePlugin is closed

[Sudheesh Katkam (JIRA)]

Sudheesh Katkam created DRILL-4773:
--

 Summary: Close all HiveMetaStoreClients when HiveStoragePlugin is 
closed
 Key: DRILL-4773
 URL: https://issues.apache.org/jira/browse/DRILL-4773
 Project: Apache Drill
  Issue Type: Bug
  Components: Storage - Hive
Reporter: Sudheesh Katkam


HiveSchemaFactory has connections to Hive metastore that are not closed when 
the HiveStoragePlugin is closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hangout Starting Now

[Sudheesh Katkam]

Hangout starting now: 
https://hangouts.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc 


Join us!

Thank you,
Sudheesh

Suggestions for Hangout Topics for 06/28/16

[Sudheesh Katkam]

If you have any suggestions for hangout topics for tomorrow, you can add it to 
this thread. We will also ask around at the beginning of the hangout for any 
topics. The goal is to try to cover whatever possible during the 1 hr.

Thank you,
Sudheesh

Re: unit test failures on master

[Sudheesh Katkam]

I am seeing unit failures as well. I suspect the last commit 
(https://github.com/apache/drill/commits/master 
).

Tests in error:
  TestJdbcMetadata.schemasWithConditions:57->JdbcTestActionBase.testAction:75 » 
DrillRuntime
  TestJdbcMetadata.tablesWithConditions:75->JdbcTestActionBase.testAction:75 » 
DrillRuntime
  TestJdbcMetadata.columnsWithConditions:93->JdbcTestActionBase.testAction:75 » 
DrillRuntime
  TestJdbcQuery.testIntegerLiteral:139 » SQL FUNCTION ERROR: Failure reading 
Fun...
  TestJdbcQuery.testSimilarNotSimilar:127 » SQL FUNCTION ERROR: Failure reading 
...
  TestJdbcQuery.testLikeNotLike:115 » SQL FUNCTION ERROR: Failure reading 
Functi...
  TestJdbcQuery.testLengthUTF8VarCharInput:353 » SQL FUNCTION ERROR: Failure 
rea...
  TestJdbcQuery.testLeftRightReplace:341 » SQL FUNCTION ERROR: Failure reading 
F...
  
Drill2128GetColumnsDataTypeNotTypeCodeIntBugsTest.testColumn_TYPE_NAME_isString:152
 » DrillRuntime
  
Drill2288GetColumnsMetadataWhenNoRowsTest.testGetTablesZeroRowsByTableSchemaOrNameWorks:189
 » DrillRuntime
  
DatabaseMetaDataGetColumnsTest.setUpConnectionAndMetadataToCheck:286->setUpRow:175
 » DrillRuntime

Thank you,
Sudheesh

> On Jun 2, 2016, at 3:23 PM, Abdel Hakim Deneche  wrote:
> 
> I'm seeing various unit test failures on my Macbook and linux vm. Maybe
> something's wrong with my environment. Can someone else run run a full
> build and see if anything fails ?
> 
> Thanks
> 
> -- 
> 
> Abdelhakim Deneche
> 
> Software Engineer
> 
>  
> 
> 
> Now Available - Free Hadoop On-Demand Training
> 

Hangout Topics For 05/31/16

[Sudheesh Katkam]

Hey y’all,

Let’s use this thread to pre-populate a list of topics to discuss on Tuesday’s 
hangout (05/31/16), so people can attend if they are interested in the 
mentioned topics. I will also collect topics at the beginning of the hangout.

+ DRILL-4280: Kerberos Authentication (Sudheesh)

Thank you,
Sudheesh