Re: Code Review Requested for DRILL-4858

[Parth Chandra]

Hi Jean-Claude,

 Thanks for the PR. I started taking a look but will have to spend some
time on making sure the evaluationVisitor change is good.

 Also, it looks like you might have pushed another commit to the same
branch and the PR has got an unrelated change. I'll assume that
(acd08439873f935b5eabebe32dae4af87b9e97db) is the relevant commit.

Parth

On Wed, Sep 14, 2016 at 8:02 PM, Jean-Claude Cote  wrote:

> Please review DRILL-4858
>
> On Tue, Sep 6, 2016 at 6:49 PM, Jean-Claude Cote  wrote:
>
> > ok let me know when you do so.
> >
> > On Fri, Sep 2, 2016 at 12:03 AM, Jinfeng Ni  wrote:
> >
> >> Thanks for submitting the PR for that issue.  I'll take a look at the
> PR.
> >>
> >> On Thu, Sep 1, 2016 at 6:36 PM, Jean-Claude Cote 
> >> wrote:
> >> > Please
> >>
> >
> >
>

Re: Code Review Requested for DRILL-4858

[Jean-Claude Cote]

Please review DRILL-4858

On Tue, Sep 6, 2016 at 6:49 PM, Jean-Claude Cote  wrote:

> ok let me know when you do so.
>
> On Fri, Sep 2, 2016 at 12:03 AM, Jinfeng Ni  wrote:
>
>> Thanks for submitting the PR for that issue.  I'll take a look at the PR.
>>
>> On Thu, Sep 1, 2016 at 6:36 PM, Jean-Claude Cote 
>> wrote:
>> > Please
>>
>
>

Re: Proposed changes for DRILL-3178

[Parth Chandra]

Hi Francois,

 I've assigned the issue to you. You don't need to have the issue assigned
to you to submit a patch though, so in future don't let that hold you back
:)

Parth

On Wed, Sep 14, 2016 at 12:05 PM, François Méthot 
wrote:

> Hi Zelaine,
>
>   I don't have the assign button available. I must be missing some
> privilege.
>
> This is the only group I am part of:
> Groups: jira-users
>
> Thanks for your quick reply
>
> On Wed, Sep 14, 2016 at 2:57 PM, Zelaine Fong  wrote:
>
> > Francois,
> >
> > Yes, feel free to assign the Jira to yourself and post a pull request.
> >
> > -- Zelaine
> >
> > On Wed, Sep 14, 2016 at 11:49 AM, François Méthot 
> > wrote:
> >
> > > Hi,
> > >
> > >   I have on my local repo a fix for
> > >
> > > https://issues.apache.org/jira/browse/DRILL-3178
> > >  csv reader should allow newlines inside quotes
> > >
> > > Can I be assigned this ticket so I can submit my proposed change?
> > >
> > > Francois
> > >
> >
>

[jira] [Resolved] (DRILL-3040) Accountor drill.exec.memory.enable_frag_limit not defaulted normally

[Paul Rogers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Rogers resolved DRILL-3040.

Resolution: Invalid

Rendered obsolete in the current memory allocator.

> Accountor drill.exec.memory.enable_frag_limit not defaulted normally
> 
>
> Key: DRILL-3040
> URL: https://issues.apache.org/jira/browse/DRILL-3040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Reporter: Daniel Barclay
> Fix For: Future
>
>
> Defaulting of drill.exec.memory.enable_frag_limit is implemented "manually" 
> in Accountor (by catching ConfigException) rather than by using the normal 
> configuration file hierarchy (being defined in the base file).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: ZK lost connectivity issue on large cluster

[Sudheesh Katkam]

Hi Francois,

Few questions:
+ How many zookeeper servers in the quorum?
+ What is the load on atsqa4-133.qa.lab when this happens? Any other 
applications running on that node? How many threads is the Drill process using?
+ When running the same query on 12 nodes, is the data size same?
+ Can you share the query profile?

This may not be the right thing to do, but for now, If the cluster is heavily 
loaded, increase the zk timeout.

Thank you,
Sudheesh

> On Sep 14, 2016, at 11:53 AM, François Méthot  wrote:
> 
> We are running 1.7.
> The log were taken from the jira tickets.
> 
> We will try out 1.8 soon.
> 
> 
> 
> 
> On Wed, Sep 14, 2016 at 2:52 PM, Chun Chang  wrote:
> 
>> Looks like you are running 1.5. I believe there are some work done in that
>> area and the newer release should behave better.
>> 
>> On Wed, Sep 14, 2016 at 11:43 AM, François Méthot 
>> wrote:
>> 
>>> Hi,
>>> 
>>>  We are trying to find a solution/workaround to issue:
>>> 
>>> 2016-01-28 16:36:14,367 [Curator-ServiceCache-0] ERROR
>>> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: ForemanException:
>>> One more more nodes lost connectivity during query.  Identified nodes
>>> were [atsqa4-133.qa.lab:31010].
>>> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
>>> ForemanException: One more more nodes lost connectivity during query.
>>> Identified nodes were [atsqa4-133.qa.lab:31010].
>>>at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.
>>> close(Foreman.java:746)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> processEvent(Foreman.java:858)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> processEvent(Foreman.java:790)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateSwitch.
>>> moveToState(Foreman.java:792)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman.moveToState(
>>> Foreman.java:909)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman.access$2700(
>>> Foreman.java:110)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>at org.apache.drill.exec.work.foreman.Foreman$StateListener.
>>> moveToState(Foreman.java:1183)
>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>> 
>>> 
>>> DRILL-4325  
>>> ForemanException:
>>> One or more nodes lost connectivity during query
>>> 
>>> 
>>> 
>>> Any one experienced this issue ?
>>> 
>>> It happens when running query involving many parquet files on a cluster
>> of
>>> 200 nodes. Same query on a smaller cluster of 12 nodes runs fine.
>>> 
>>> It is not caused by garbage collection, (checked on both ZK node and the
>>> involved drill bit).
>>> 
>>> Negotiated max session timeout is 40 seconds.
>>> 
>>> The sequence seems:
>>> - Drill Query begins, using an existing ZK session.
>>> - Drill Zk session timeouts
>>>  - perhaps it was writing something that took too long
>>> - Drill attempts to renew session
>>>   - drill believes that the write operation failed, so it attempts
>> to
>>> re-create the zk node, which trigger another exception.
>>> 
>>> We are open to any suggestion. We will report any finding.
>>> 
>>> Thanks
>>> Francois
>>> 
>> 

[jira] [Created] (DRILL-4893) Define in Drill-override.conf a common DFS root directory

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-4893:
--

 Summary: Define in Drill-override.conf a common DFS root directory
 Key: DRILL-4893
 URL: https://issues.apache.org/jira/browse/DRILL-4893
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Paul Rogers
Priority: Minor


Drill-on-YARN and Dynamic UDFs are two Drill features that write files to the 
distributed file system (DFS) such as HDFS. These files are configuration-style 
files (Drill software for DoY, UDF jars for Dynamic UDFs.) Likely other Drill 
features will also use DFS for storage, perhaps for logs, metadata, etc.

At present, each feature defines its own DFS storage path. An administrator who 
wants to define a DFS location must know of, and modify, each of these 
locations.

The proposal here is to define a single Drill root directory property. Then, 
define all other DFS locations relative to the root. This allows the admin to 
relocate all Drill files by changing a single property. (Of course, this change 
should be made at install time, not after Drill has written files, unless the 
files themselves are moved.)

For example (using made-up property names):

{code}
drill.dfs: “/user/drill”
drill.yarn.dfs: “${drill.dfs}/doy”
drill.exec.udf.directory.base: “${drill.dfs}/udf”
drill.exec.udf.directory.staging: “${drill.exec.udf.directory.base}/staging”
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Proposed changes for DRILL-3178

[François Méthot]

Hi Zelaine,

  I don't have the assign button available. I must be missing some
privilege.

This is the only group I am part of:
Groups: jira-users

Thanks for your quick reply

On Wed, Sep 14, 2016 at 2:57 PM, Zelaine Fong  wrote:

> Francois,
>
> Yes, feel free to assign the Jira to yourself and post a pull request.
>
> -- Zelaine
>
> On Wed, Sep 14, 2016 at 11:49 AM, François Méthot 
> wrote:
>
> > Hi,
> >
> >   I have on my local repo a fix for
> >
> > https://issues.apache.org/jira/browse/DRILL-3178
> >  csv reader should allow newlines inside quotes
> >
> > Can I be assigned this ticket so I can submit my proposed change?
> >
> > Francois
> >
>

[jira] [Created] (DRILL-4892) Swift Documentation

[Matt Keranen (JIRA)]

Matt Keranen created DRILL-4892:
---

 Summary: Swift Documentation
 Key: DRILL-4892
 URL: https://issues.apache.org/jira/browse/DRILL-4892
 Project: Apache Drill
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.8.0, 1.6.0
Reporter: Matt Keranen


The Drill FAQ (https://drill.apache.org/faq/), suggest Swift is a datasource:

"Cloud storage: Amazon S3, Google Cloud Storage, Azure Blog Storage, Swift"

However there appears to be no documentation (?)

Swift specific docs would be very useful. We have a large Swift installation 
and using Drill over files in it would be a valuable feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4891) Use field width when available for computing estimated memory costing

[Aman Sinha (JIRA)]

Aman Sinha created DRILL-4891:
-

 Summary: Use field width when available for computing estimated 
memory costing
 Key: DRILL-4891
 URL: https://issues.apache.org/jira/browse/DRILL-4891
 Project: Apache Drill
  Issue Type: Improvement
  Components: Query Planning & Optimization
Affects Versions: 1.8.0
Reporter: Aman Sinha


Drill currently uses an average field width (in addition to other parameters) 
for computing estimated memory cost for hash and sort based operations.  The 
assumption is that the field is of ANY type.  However, when a view with 
appropriate CAST is used, the relevant data type is available and we should 
leverage the correct field width for the costing calculations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r78798093
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -246,28 +154,80 @@ protected void handle(UserClientConnectionImpl 
connection, int rpcType, ByteBuf
   public class UserClientConnectionImpl extends RemoteConnection 
implements UserClientConnection {
 
 private UserSession session;
+private SaslServer saslServer;
+private RequestHandler currentHandler;
+private UserToBitHandshake inbound;
 
 public UserClientConnectionImpl(SocketChannel channel) {
   super(channel, "user client");
+  currentHandler = authFactory == null ? handler : new 
UserServerAuthenticationHandler(handler);
 }
 
 void disableReadTimeout() {
   getChannel().pipeline().remove(BasicServer.TIMEOUT_HANDLER);
 }
 
-void setUser(final UserToBitHandshake inbound) throws IOException {
+void setHandshake(final UserToBitHandshake inbound) throws IOException 
{
+  this.inbound = inbound;
+}
+
+void initSaslServer(final String mechanismName, final Map 
properties)
+throws IllegalStateException, SaslException {
+  if (saslServer != null) {
+throw new IllegalStateException("SASL server already 
initialized.");
+  }
+  this.saslServer = authFactory.getMechanism(mechanismName)
--- End diff --

It should be the responsibility of class AuthenticationMechanismFactory to 
decide what mechanism to provide. Right? So AuthenticationMechanismFactory may 
also need 'properties' to decide what mechanism to ask for. E.g. Some company 
may want connections from out of VPN or in VPN to use different mechanisms.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r78791984
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/plain/PlainServer.java
 ---
@@ -0,0 +1,174 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security.plain;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.PasswordCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.AuthorizeCallback;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslException;
+import javax.security.sasl.SaslServer;
+import javax.security.sasl.SaslServerFactory;
+import java.io.IOException;
+import java.security.Provider;
+import java.util.Map;
+
+/**
+ * Plain SaslServer implementation. See https://tools.ietf.org/html/rfc4616
+ */
+public class PlainServer implements SaslServer {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PlainServer.class);
+
+  public static class PlainServerFactory implements SaslServerFactory {
+
+@Override
+public SaslServer createSaslServer(final String mechanism, final 
String protocol, final String serverName,
+   final Map props, final 
CallbackHandler cbh)
+throws SaslException {
+  return "PLAIN".equals(mechanism) ?
+  props == null || 
"false".equals(props.get(Sasl.POLICY_NOPLAINTEXT)) ?
+  new PlainServer(cbh) :
+  null :
+  null;
+}
+
+@Override
+public String[] getMechanismNames(final Map props) {
+  return props == null || 
"false".equals(props.get(Sasl.POLICY_NOPLAINTEXT)) ?
+  new String[]{"PLAIN"} :
+  new String[0];
+}
+  }
+
+  @SuppressWarnings("serial")
+  public static class PlainServerProvider extends Provider {
+
+public PlainServerProvider() {
+  super("PlainServer", 1.0, "PLAIN SASL Server Provider");
+  put("SaslServerFactory.PLAIN", PlainServerFactory.class.getName());
+}
+  }
+
+  private CallbackHandler cbh;
+  private boolean completed;
+  private String authorizationID;
+
+  PlainServer(final CallbackHandler cbh) throws SaslException {
+if (cbh == null) {
+  throw new SaslException("PLAIN: A callback handler must be 
specified.");
+}
+this.cbh = cbh;
+  }
+
+  @Override
+  public String getMechanismName() {
+return "PLAIN";
+  }
+
+  @Override
+  public byte[] evaluateResponse(byte[] response) throws SaslException {
+if (completed) {
+  throw new IllegalStateException("PLAIN authentication already 
completed");
+}
+
+if (response == null) {
+  throw new SaslException("Received null response");
+}
+
+completed = true;
--- End diff --

'completed' should be set at the end of the whole authentication since it 
is the critical flag to decide if the user was authenticated successfully.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r78791283
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserProtos.java ---
@@ -47,157 +47,165 @@ public static void registerAllExtensions(
  */
 GOODBYE(2, 2),
 /**
+ * SASL_MESSAGE = 24;
+ *
+ * 
+ * user to bit and bit to user
+ * 
+ */
+SASL_MESSAGE(3, 24),
--- End diff --

What is the reason we inject SASL_MESSAGE here and we have to update all 
definitions below?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #578: DRILL-4280: Kerberos Authentication

[chunhui-shi]

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r78790827
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/plain/PlainServer.java
 ---
@@ -0,0 +1,174 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security.plain;
--- End diff --

Is there any reason we have to implement our own PlainServer? What about 
hadoop's SaslPlainServer?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: DRILL JDBC Driver setFetchSize

[Sudheesh Katkam]

Hi Sudip,

fetchSize is number of records to fetch per network call from the server (to 
populate a ResultSet), not the same as LIMIT clause. See [1].

Currently, the client does not advertise this to the server, and the server 
does not have this capability.

Thank you,
Sudheesh

[1] 
https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setFetchSize(int)
 


> On Sep 13, 2016, at 11:48 PM, Sudip Mukherjee  
> wrote:
> 
> Hi,
> 
> Should the setFetchSize work similar to applying LIMIT clause with drill JDBC 
> driver? I've set setFetchSize to 
> java.sql.Statement
>  but the query still gets records beyond the fetchSIze.
> 
> Thanks,
> Sudip

Re: Nested/n-dimensional Arrays in Mongo/Json and Drill

[Pradeeban Kathiravelu]

Hi Sudeesh,
Probably I was not clear in my previous email. The error is due to
accessing an array from the SELECT directly, which is not supported by
Drill. My question was, how to overcome this hurdle without using the
performance-hungry FLATTEN keyword or using the array indices (impractical
as the data is not structured and we do not know the number of elements,
and it is going to be a lot and a 3-D array). I am planning to use
Protobufs with Drill to serialize and deserialize the data.

Anyway, since you asked, given below is the error logs when accessing the
nested array element using SELECT. (The entire query and the sample data
were given in the previous email).


Error: SYSTEM ERROR: NullPointerException

Fragment 0:0

[Error Id: 98390161-00e7-4982-b40b-171c8f1029c3 on llovizna:31010]

  (java.lang.NullPointerException) null
org.apache.drill.exec.vector.complex.impl.MapOrListWriterImpl.list():73
org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap():112
org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap():112
org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap():139
org.apache.drill.exec.store.bson.BsonRecordReader.write():75
org.apache.drill.exec.store.mongo.MongoRecordReader.next():186
org.apache.drill.exec.physical.impl.ScanBatch.next():191
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51

org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():94
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.record.AbstractRecordBatch.next():119
org.apache.drill.exec.record.AbstractRecordBatch.next():109
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51

org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():162
org.apache.drill.exec.physical.impl.BaseRootExec.next():104

org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():81
org.apache.drill.exec.physical.impl.BaseRootExec.next():94
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():257
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():251
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1657
org.apache.drill.exec.work.fragment.FragmentExecutor.run():251
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1142
java.util.concurrent.ThreadPoolExecutor$Worker.run():617
java.lang.Thread.run():744 (state=,code=0)


Thank you.
Regards,
Pradeeban.

On Tue, Sep 13, 2016 at 8:34 PM, Sudheesh Katkam 
wrote:

> Hi Pradeeban,
>
> Can you post the detailed error message?
>
> First set the option:
> > SET `exec.errors.verbose` = true;
>
> And then run the query. The detailed output will point us to where the
> error occurred.
>
> Thank you,
> Sudheesh
>
> > On Sep 12, 2016, at 10:47 AM, Pradeeban Kathiravelu <
> kk.pradee...@gmail.com> wrote:
> >
> > Hi,
> > I have a complex json data stored in Mongo.
> >
> > The data has nested arrays as shown below .
> >
> > here when I try to access the geometry field in select queries, the Drill
> > query fails.
> >
> > select camic._id, camic.type, camic.parent_id, camic.randval,
> > camic.creation_date, camic.object_type, camic.x, camic.y,
> camic.normalized,
> > camic.bbox, camic.geometry, camic.footprint, camic.properties,
> > camic.provenance, camic.submit_date from mongo.CAMICROSCOPE.`testUAIM2`
> as
> > camic WHERE ((camic.provenance.image.case_id =
> 'TCGA-02-0001-01Z-00-DX1')
> > AND (camic.provenance.analysis.execution_id  = 'tammy-test:7') AND
> > (camic.footprint >= 800) AND (camic.x >= 0) AND (camic.x <=1) AND
> (camic.y
> >> = 0) AND (camic.y <= 1));
> >
> > would fail with a below error:
> >
> >
> >
> >
> >
> > *Error: SYSTEM ERROR: NullPointerExceptionFragment 0:0[Error Id:
> > 8cd950af-91fa-4cf8-865b-265f227c8e87 on llovizna:31010] (state=,code=0)*
> >
> > However, after removing the geometry from the select query, it would work
> > just fine.
> >
> > select 

[jira] [Created] (DRILL-4890) Right outer join fails

[Kathiresan Selvaraj (JIRA)]

Kathiresan Selvaraj created DRILL-4890:
--

 Summary: Right outer join fails 
 Key: DRILL-4890
 URL: https://issues.apache.org/jira/browse/DRILL-4890
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Data Types, Execution - Flow
Affects Versions: 1.8.0, 1.6.0
Reporter: Kathiresan Selvaraj


Hi,  A Query with right outer join fails while the inner and left outer joins 
work for the same data. I've replicated the issue with some simple data here 
and this happens in both 1.6.0 and 1.8.0

*Json file 1: data.json*

{ "name": "Jim","city" : [1,2]}

*Json file 2: cities.json*

{id:1,name:"Sendurai"}
{id:2,name:"NYC"}

*Queries that work:*
1.  select a.name,a.city,b.id,b.name from dfs.tmp.`data.json` a left outer join 
dfs.tmp.`cities.json` b on a.city\[0]=b.id

2. select a.name,a.city,b.id,b.name from dfs.tmp.`data.json` a join 
dfs.tmp.`cities.json` b on a.city\[0]=b.id

*Query that fails:*

select a.name,a.city,b.id,b.name from dfs.tmp.`data.json` a right outer join 
dfs.tmp.`cities.json` b on a.city\[0]=b.id

*On the server side, i see below error trace :*

java.lang.IllegalStateException: QueryDataBatch was released twice.
at 
org.apache.drill.exec.rpc.user.QueryDataBatch.release(QueryDataBatch.java:56) 
\[drill-java-exec-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.user.QueryResultHandler.batchArrived(QueryResultHandler.java:167)
 \[drill-java-exec-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110) 
~\[drill-java-exec-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
 ~\[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
 ~\[drill-rpc-1.6.0.jar:1.6.0]
at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:67) 
~\[drill-rpc-1.6.0.jar:1.6.0]
at org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:374) 
~\[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
 \[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:252) 
\[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123) 
\[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:285) 
\[drill-rpc-1.6.0.jar:1.6.0]
at 
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:257) 
\[drill-rpc-1.6.0.jar:1.6.0]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4889) throw NPE or AssertionError in FindPartitionConditions when running some SQLs on partition table

[godfrey he (JIRA)]

godfrey he created DRILL-4889:
-

 Summary: throw NPE or AssertionError in FindPartitionConditions 
when running some SQLs on partition table
 Key: DRILL-4889
 URL: https://issues.apache.org/jira/browse/DRILL-4889
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning & Optimization
Affects Versions: 1.8.0
Reporter: godfrey he


when I ran some SQLs on partition table, I got NPE or AssertionError in 
FindPartitionConditions 

partition_table is a partition table
partition_column is the partition column
none_partition_column is a normal column, not the partition column

SQL 1: select * from partition_table where none_partition_column like '%test%'; 
and also
SQL 2:  select * from partition_table where case when partition_column = '2000' 
then none_partition_column = 'test' end;

Caused by: java.lang.NullPointerException: null
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions$OpState.access$000(FindPartitionConditions.java:63)
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.analyzeCall(FindPartitionConditions.java:315)
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.visitCall(FindPartitionConditions.java:260)
 
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.visitCall(FindPartitionConditions.java:46)


SQL 3: select * from partition_table where partition_column = 'a' or 
(partition_column = 'b' and partition_column = '2000')

Caused by: java.lang.AssertionError: null
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.popOpStackAndBuildFilter(FindPartitionConditions.java:198)
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.analyzeCall(FindPartitionConditions.java:335)
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.visitCall(FindPartitionConditions.java:260)
at 
org.apache.drill.exec.planner.logical.partition.FindPartitionConditions.visitCall(FindPartitionConditions.java:46)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

DRILL JDBC Driver setFetchSize

[Sudip Mukherjee]

Hi,

Should the setFetchSize work similar to applying LIMIT clause with drill JDBC 
driver? I've set setFetchSize to 
java.sql.Statement
 but the query still gets records beyond the fetchSIze.

Thanks,
Sudip