[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460179#comment-15460179
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429753
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -308,33 +312,57 @@ public BitToUserHandshake 
getHandshakeResponse(UserToBitHandshake inbound) throw
 connection.setHandshake(inbound);
 
 try {
+  // TODO(SUDHEESH): MUST FIX THIS VERSION CHECK FIRST BEFORE THE 
CHECK BELOW
   if (inbound.getRpcVersion() != UserRpcConfig.RPC_VERSION) {
 final String errMsg = String.format("Invalid rpc version. 
Expected %d, actual %d.",
 UserRpcConfig.RPC_VERSION, inbound.getRpcVersion());
 
 return handleFailure(respBuilder, 
HandshakeStatus.RPC_VERSION_MISMATCH, errMsg, null);
   }
 
-  if (authenticator != null) {
-try {
-  String password = "";
-  final UserProperties props = inbound.getProperties();
-  for (int i = 0; i < props.getPropertiesCount(); i++) {
-Property prop = props.getProperties(i);
-if (UserSession.PASSWORD.equalsIgnoreCase(prop.getKey())) {
-  password = prop.getValue();
-  break;
+  connection.setHandshake(inbound);
+
+  if (authFactory != null) {
+if (inbound.getRpcVersion() <= 5) { // for backward 
compatibility <= 1.8
+  final String userName = 
inbound.getCredentials().getUserName();
+  if (logger.isTraceEnabled()) {
+logger.trace("User {} on connection {} is using an older 
client (Drill version <= 1.8).",
+userName, connection.getRemoteAddress());
+  }
+  try {
+String password = "";
+final UserProperties props = inbound.getProperties();
+for (int i = 0; i < props.getPropertiesCount(); i++) {
+  Property prop = props.getProperties(i);
+  if 
(UserSession.PASSWORD.equalsIgnoreCase(prop.getKey())) {
+password = prop.getValue();
+break;
+  }
+}
+final PlainMechanism plainMechanism = 
authFactory.getPlainMechanism();
+if (plainMechanism == null) {
+  throw new UserAuthenticationException("The server no 
longer supports username/password" +
+  " based authentication. Please talk to your system 
administrator.");
 }
+plainMechanism.getAuthenticator().authenticate(userName, 
password);
+connection.changeHandlerTo(handler);
+connection.finalizeSession(userName);
+respBuilder.setStatus(HandshakeStatus.SUCCESS);
--- End diff --

Would it be useful to add log the successful mechanism (where we have 
`respBuilder.setStatus(HandshakeStatus.SUCCESS);`)?


> Kerberos Authentication
> ---
>
> Key: DRILL-4280
> URL: https://issues.apache.org/jira/browse/DRILL-4280
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Keys Botzum
>Assignee: Sudheesh Katkam
>  Labels: security
>
> Drill should support Kerberos based authentication from clients. This means 
> that both the ODBC and JDBC drivers as well as the web/REST interfaces should 
> support inbound Kerberos. For Web this would most likely be SPNEGO while for 
> ODBC and JDBC this will be more generic Kerberos.
> Since Hive and much of Hadoop supports Kerberos there is a potential for a 
> lot of reuse of ideas if not implementation.
> Note that this is related to but not the same as 
> https://issues.apache.org/jira/browse/DRILL-3584 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460166#comment-15460166
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429640
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -308,33 +312,57 @@ public BitToUserHandshake 
getHandshakeResponse(UserToBitHandshake inbound) throw
 connection.setHandshake(inbound);
 
 try {
+  // TODO(SUDHEESH): MUST FIX THIS VERSION CHECK FIRST BEFORE THE 
CHECK BELOW
   if (inbound.getRpcVersion() != UserRpcConfig.RPC_VERSION) {
 final String errMsg = String.format("Invalid rpc version. 
Expected %d, actual %d.",
 UserRpcConfig.RPC_VERSION, inbound.getRpcVersion());
 
 return handleFailure(respBuilder, 
HandshakeStatus.RPC_VERSION_MISMATCH, errMsg, null);
   }
 
-  if (authenticator != null) {
-try {
-  String password = "";
-  final UserProperties props = inbound.getProperties();
-  for (int i = 0; i < props.getPropertiesCount(); i++) {
-Property prop = props.getProperties(i);
-if (UserSession.PASSWORD.equalsIgnoreCase(prop.getKey())) {
-  password = prop.getValue();
-  break;
+  connection.setHandshake(inbound);
+
+  if (authFactory != null) {
+if (inbound.getRpcVersion() <= 5) { // for backward 
compatibility <= 1.8
--- End diff --

Can we use something like MIN_COMPATIBLE_RPC_VERSION for 5?


> Kerberos Authentication
> ---
>
> Key: DRILL-4280
> URL: https://issues.apache.org/jira/browse/DRILL-4280
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Keys Botzum
>Assignee: Sudheesh Katkam
>  Labels: security
>
> Drill should support Kerberos based authentication from clients. This means 
> that both the ODBC and JDBC drivers as well as the web/REST interfaces should 
> support inbound Kerberos. For Web this would most likely be SPNEGO while for 
> ODBC and JDBC this will be more generic Kerberos.
> Since Hive and much of Hadoop supports Kerberos there is a potential for a 
> lot of reuse of ideas if not implementation.
> Note that this is related to but not the same as 
> https://issues.apache.org/jira/browse/DRILL-3584 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460161#comment-15460161
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429619
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -308,33 +312,57 @@ public BitToUserHandshake 
getHandshakeResponse(UserToBitHandshake inbound) throw
 connection.setHandshake(inbound);
--- End diff --

Why do we call `connection.setHandshake(inbound);` again below?


> Kerberos Authentication
> ---
>
> Key: DRILL-4280
> URL: https://issues.apache.org/jira/browse/DRILL-4280
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Keys Botzum
>Assignee: Sudheesh Katkam
>  Labels: security
>
> Drill should support Kerberos based authentication from clients. This means 
> that both the ODBC and JDBC drivers as well as the web/REST interfaces should 
> support inbound Kerberos. For Web this would most likely be SPNEGO while for 
> ODBC and JDBC this will be more generic Kerberos.
> Since Hive and much of Hadoop supports Kerberos there is a potential for a 
> lot of reuse of ideas if not implementation.
> Note that this is related to but not the same as 
> https://issues.apache.org/jira/browse/DRILL-3584 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460148#comment-15460148
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429494
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java ---
@@ -308,33 +312,57 @@ public BitToUserHandshake 
getHandshakeResponse(UserToBitHandshake inbound) throw
 connection.setHandshake(inbound);
 
 try {
+  // TODO(SUDHEESH): MUST FIX THIS VERSION CHECK FIRST BEFORE THE 
CHECK BELOW
--- End diff --

Does this need to be fixed? If not, then we should remove the comment.


> Kerberos Authentication
> ---
>
> Key: DRILL-4280
> URL: https://issues.apache.org/jira/browse/DRILL-4280
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Keys Botzum
>Assignee: Sudheesh Katkam
>  Labels: security
>
> Drill should support Kerberos based authentication from clients. This means 
> that both the ODBC and JDBC drivers as well as the web/REST interfaces should 
> support inbound Kerberos. For Web this would most likely be SPNEGO while for 
> ODBC and JDBC this will be more generic Kerberos.
> Since Hive and much of Hadoop supports Kerberos there is a potential for a 
> lot of reuse of ideas if not implementation.
> Note that this is related to but not the same as 
> https://issues.apache.org/jira/browse/DRILL-3584 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460145#comment-15460145
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429451
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/AuthenticationMechanismFactory.java
 ---
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security;
+
+import com.google.common.base.Function;
+import com.google.common.base.Strings;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Sets;
+import org.apache.drill.common.AutoCloseables;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.map.CaseInsensitiveMap;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.apache.drill.exec.exception.DrillbitStartupException;
+import org.apache.drill.exec.rpc.security.kerberos.KerberosMechanism;
+import org.apache.drill.exec.rpc.security.plain.PlainMechanism;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticator;
+import org.apache.drill.exec.rpc.user.security.UserAuthenticatorFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class AuthenticationMechanismFactory implements AutoCloseable {
+  private static final org.slf4j.Logger logger =
+  
org.slf4j.LoggerFactory.getLogger(AuthenticationMechanismFactory.class);
+
+  private final Map mechanisms = 
CaseInsensitiveMap.newHashMapWithExpectedSize(5);
+
+  @SuppressWarnings("unchecked")
+  public AuthenticationMechanismFactory(final ScanResult scan, final 
DrillConfig config,
+final List 
configuredMechanisms)
+  throws DrillbitStartupException {
+logger.debug("Configuring authentication mechanisms: {}", 
configuredMechanisms);
+// transform all names to uppercase
+final Set configuredMechanismsSet = 
Sets.newHashSet(Iterators.transform(configuredMechanisms.iterator(),
+new Function() {
+  @Nullable
+  @Override
+  public String apply(@Nullable String input) {
+return input == null ? null : input.toUpperCase();
+  }
+}));
+
+// First, load Drill provided out-of-box mechanisms
+if (configuredMechanismsSet.contains(PlainMechanism.SIMPLE_NAME)) {
+  logger.trace("Plain mechanism enabled.");
+  // instantiated here, but closed in PlainMechanism#close
+  final UserAuthenticator userAuthenticator = 
UserAuthenticatorFactory.createAuthenticator(config, scan);
+  mechanisms.put(PlainMechanism.SIMPLE_NAME, new 
PlainMechanism(userAuthenticator));
+  configuredMechanismsSet.remove(PlainMechanism.SIMPLE_NAME);
+}
+
+if (configuredMechanismsSet.contains(KerberosMechanism.SIMPLE_NAME)) {
+  logger.trace("Kerberos mechanism enabled.");
+  final String servicePrincipal = 
config.getString("drill.exec.security.auth.principal");
+  final String keytab = 
config.getString("drill.exec.security.auth.keytab");
+
+  try { // Kerberos mechanism requires a service to login
+final Configuration conf = new Configuration();
+conf.set(CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION, 
"KERBEROS");
+// To parse non-typical principal name, uncomment 

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460141#comment-15460141
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77429414
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/security/plain/PlainServer.java
 ---
@@ -0,0 +1,174 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.security.plain;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.PasswordCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.AuthorizeCallback;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslException;
+import javax.security.sasl.SaslServer;
+import javax.security.sasl.SaslServerFactory;
+import java.io.IOException;
+import java.security.Provider;
+import java.util.Map;
+
+/**
+ * Plain SaslServer implementation. See https://tools.ietf.org/html/rfc4616
+ */
+public class PlainServer implements SaslServer {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(PlainServer.class);
+
+  public static class PlainServerFactory implements SaslServerFactory {
+
+@Override
+public SaslServer createSaslServer(final String mechanism, final 
String protocol, final String serverName,
+   final Map props, final 
CallbackHandler cbh)
+throws SaslException {
+  return "PLAIN".equals(mechanism) ?
+  props == null || 
"false".equals(props.get(Sasl.POLICY_NOPLAINTEXT)) ?
+  new PlainServer(cbh) :
+  null :
+  null;
+}
+
+@Override
+public String[] getMechanismNames(final Map props) {
+  return props == null || 
"false".equals(props.get(Sasl.POLICY_NOPLAINTEXT)) ?
+  new String[]{"PLAIN"} :
+  new String[0];
+}
+  }
+
+  @SuppressWarnings("serial")
+  public static class PlainServerProvider extends Provider {
+
+public PlainServerProvider() {
+  super("PlainServer", 1.0, "PLAIN SASL Server Provider");
+  put("SaslServerFactory.PLAIN", PlainServerFactory.class.getName());
+}
+  }
+
+  private CallbackHandler cbh;
+  private boolean completed;
+  private String authorizationID;
+
+  PlainServer(final CallbackHandler cbh) throws SaslException {
+if (cbh == null) {
+  throw new SaslException("PLAIN: A callback handler must be 
specified.");
+}
+this.cbh = cbh;
+  }
+
+  @Override
+  public String getMechanismName() {
+return "PLAIN";
+  }
+
+  @Override
+  public byte[] evaluateResponse(byte[] response) throws SaslException {
+if (completed) {
+  throw new IllegalStateException("PLAIN authentication already 
completed");
+}
+
+if (response == null) {
+  throw new SaslException("Received null response");
+}
+
+completed = true;
+final String payload;
+try {
+  payload = new String(response, "UTF-8");
+} catch (final Exception e) {
+  throw new SaslException("Received corrupt response", e);
+}
+
+// Separator defined in PlainClient is 0
+// three parts: [ authorizationID, authenticationID, password ]
+final String[] parts = payload.split("\u", 3);
+if (parts.length != 3) {
+  throw new SaslException("Received corrupt response. Expected 3 
parts, but received "
+  + parts.length);
+}
+if (parts[0].isEmpty()) {
+  parts[0] = parts[1]; // 

[jira] [Commented] (DRILL-4872) NPE from CTAS partitioned by a projected casted null

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459989#comment-15459989
 ] 

Boaz Ben-Zvi commented on DRILL-4872:
-

Likely related ...

> NPE from CTAS partitioned by a projected casted null
> 
>
> Key: DRILL-4872
> URL: https://issues.apache.org/jira/browse/DRILL-4872
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Boaz Ben-Zvi
>  Labels: NPE
> Fix For: Future
>
>
> Extracted from DRILL-3898 : Running the same test case on a smaller table ( 
> store_sales.dat from TPCDS SF 1) has no space issues, but there is a Null 
> Pointer Exception from the projection:
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.doEval(ProjectorTemplate.java:49)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> A simplified version of the test case:
> 0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
> case when columns[8] = '' then cast(null as varchar(10)) else cast(columns[8] 
> as varchar(10)) end as x FROM 
> dfs.`/Users/boazben-zvi/data/store_sales/store_sales.dat`;
> Error: SYSTEM ERROR: NullPointerException
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4872) NPE from CTAS partitioned by a projected casted null

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459987#comment-15459987
 ] 

Boaz Ben-Zvi commented on DRILL-4872:
-

 And the NPE error can be recreated with the same data as in DRILL-4026:

0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
case when columns[0] = '' then cast(null as varchar(10)) else cast(columns[0] 
as varchar(10)) end as x FROM dfs.`/Users/boazben-zvi/data/abc.tbl`;
Error: SYSTEM ERROR: NullPointerException



> NPE from CTAS partitioned by a projected casted null
> 
>
> Key: DRILL-4872
> URL: https://issues.apache.org/jira/browse/DRILL-4872
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Boaz Ben-Zvi
>  Labels: NPE
> Fix For: Future
>
>
> Extracted from DRILL-3898 : Running the same test case on a smaller table ( 
> store_sales.dat from TPCDS SF 1) has no space issues, but there is a Null 
> Pointer Exception from the projection:
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.doEval(ProjectorTemplate.java:49)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> A simplified version of the test case:
> 0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
> case when columns[8] = '' then cast(null as varchar(10)) else cast(columns[8] 
> as varchar(10)) end as x FROM 
> dfs.`/Users/boazben-zvi/data/store_sales/store_sales.dat`;
> Error: SYSTEM ERROR: NullPointerException
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4872) NPE from CTAS partitioned by a projected casted null

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459977#comment-15459977
 ] 

Boaz Ben-Zvi commented on DRILL-4872:
-

Likely related - notice the similarity: below is the stack from the current 
code (for the testcase in DRILL-4026) :

Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0
at 
io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1178) 
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:115) 
~[drill-memory-base-1.8.0-SNAPSHOT.jar:4.0.27.Final]
at io.netty.buffer.DrillBuf.checkBytes(DrillBuf.java:141) 
~[drill-memory-base-1.8.0-SNAPSHOT.jar:4.0.27.Final]
at 
org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:99)
 ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at 
org.apache.drill.exec.test.generated.ProjectorGen3.doEval(ProjectorTemplate.java:49)
 ~[na:na]
at 
org.apache.drill.exec.test.generated.ProjectorGen3.projectRecords(ProjectorTemplate.java:62)
 ~[na:na]
at 
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
 ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
at 
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
 ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]

> NPE from CTAS partitioned by a projected casted null
> 
>
> Key: DRILL-4872
> URL: https://issues.apache.org/jira/browse/DRILL-4872
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Boaz Ben-Zvi
>  Labels: NPE
> Fix For: Future
>
>
> Extracted from DRILL-3898 : Running the same test case on a smaller table ( 
> store_sales.dat from TPCDS SF 1) has no space issues, but there is a Null 
> Pointer Exception from the projection:
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.doEval(ProjectorTemplate.java:49)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> A simplified version of the test case:
> 0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
> case when columns[8] = '' then cast(null as varchar(10)) else cast(columns[8] 
> as varchar(10)) end as x FROM 
> dfs.`/Users/boazben-zvi/data/store_sales/store_sales.dat`;
> Error: SYSTEM ERROR: NullPointerException
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459819#comment-15459819
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422631
  
--- Diff: exec/java-exec/src/main/codegen/includes/parserImpls.ftl ---
@@ -297,4 +297,44 @@ SqlNode SqlDescribeSchema() :
{
 return new SqlDescribeSchema(pos, schema);
}
+}
--- End diff --

Looks fine conceptually, but have someone double-check this who is more 
familiar with the details of the parser implementation.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459817#comment-15459817
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422515
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java
 ---
@@ -141,25 +141,41 @@ public void validate(final OptionValue v, final 
OptionManager manager) {
 
   public static class BooleanValidator extends TypeValidator {
 public BooleanValidator(String name, boolean def) {
-  super(name, Kind.BOOLEAN, 
OptionValue.createBoolean(OptionType.SYSTEM, name, def));
+  this(name, def, false);
--- End diff --

Looks file, but should have someone review this who is more familiar with 
this part of the code. 


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459814#comment-15459814
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422394
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/beans/Registry.java ---
@@ -0,0 +1,175 @@
+/**
--- End diff --

Not reviewed: this is a generated file.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459812#comment-15459812
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422352
  
--- Diff: protocol/src/main/java/org/apache/drill/exec/proto/beans/Jar.java 
---
@@ -0,0 +1,197 @@
+/**
--- End diff --

Not reviewed: this is a generated file.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459806#comment-15459806
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422238
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/SchemaUserBitShared.java ---
@@ -2678,4 +2678,357 @@ public static int getFieldNumber(java.lang.String 
name)
 }
 }
 
+public static final class Func
--- End diff --

Not reviewed: this is a generated file.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459808#comment-15459808
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422265
  
--- Diff: 
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java ---
@@ -20697,6 +20697,2346 @@ public Builder clearDoubleValue() {
 // @@protoc_insertion_point(class_scope:exec.shared.MetricValue)
   }
 
+  public interface FuncOrBuilder
--- End diff --

Not reviewed: this is a generated file.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459805#comment-15459805
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77422179
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
   debug: {
 return_error_for_failure_in_cancelled_fragments: false
   }
+  udf: {
+retry.times: 5,
+directory: {
+  base: "/tmp",
+  staging: ${drill.exec.zk.root}"/udf/staging",
--- End diff --

I wonder, are we trying to define two distinct things here? A ZK zNode and 
a DFS directory? If so, perhaps split them between

drill.exc.udf.dynamic
  dfs
directory props...
  zk
 zk zNodes

It is not (yet) clear exactly how these are used, so this comment may be 
missing the point if we configure only dfs locations...


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459798#comment-15459798
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421971
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
+  repeated common.MajorType major_type = 2;
+}
+
+message Jar {
+  optional string name = 1;
+  repeated Func function = 2;
+}
+
+message Registry {
--- End diff --

When do we exchange a registry of Jars? Is this stored in ZK? If so, do we 
have to worry about the total size of the resulting ZK zNode? Have we done 
performance checks to see what happens if the user has registered, say, 100 
jars of 5 functions each? Should we consider a ZK structure (if that is what 
this is for)  of the form:

.../udfs
  jar1: 
  jar2: 

That is, store the list of jars as zNodes, where each zNode defines one jar?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459795#comment-15459795
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421764
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
+  repeated common.MajorType major_type = 2;
+}
+
+message Jar {
--- End diff --

A Jar contains one or more functions?
The name is the file name? Which, presumably, the user must ensure is 
unique across statically defined UDFs (those in the class path) and dynamic 
UDFS?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459792#comment-15459792
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421640
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
--- End diff --

This is the name of the UDF class? Presumably the full name with package 
prefix?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459790#comment-15459790
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421538
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
+  repeated common.MajorType major_type = 2;
--- End diff --

What does major type mean for a function? Is this the parameter type? The 
return type? Something else? Perhaps choose a clearer name or include a comment 
to explain the meaning.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459787#comment-15459787
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421468
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
--- End diff --

Protobufs are hard to guess at without documentation; there is no code to 
provide a hint about the meaning of fields. Can you provide a description of 
how these are used?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459778#comment-15459778
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421120
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
   debug: {
 return_error_for_failure_in_cancelled_fragments: false
   }
+  udf: {
+retry.times: 5,
+directory: {
+  base: "/tmp",
--- End diff --

Is there a /tmp in DFS? If so, this would be a shared resoure. Should the 
base be /tmp/drill to keep our temp files distinct from those of other apps?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459774#comment-15459774
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77421023
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
   debug: {
 return_error_for_failure_in_cancelled_fragments: false
   }
+  udf: {
+retry.times: 5,
+directory: {
+  base: "/tmp",
+  staging: ${drill.exec.zk.root}"/udf/staging",
--- End diff --

zk.root is the root zNode in ZK; it is not a file system node in DFS.

See above comment about defining a new drill.dfs.root property.

I like how you've used the ${} substitution here.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4874) "No UserGroupInformation while generating ORC splits" - hive known issue in 1.2.0-mapr-1607 release.

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-4874:
---
Description: 
Need upgrade drill to 1.2.0-mapr-1608 hive.version where [hive issue 
HIVE-13120|https://issues.apache.org/jira/browse/HIVE-13120] is fixed.


  was:
Need upgrade drill to 1.2.0-mapr-1608 hive.version where hvie issue 
[HIVE-13120|https://issues.apache.org/jira/browse/HIVE-13120] is fixed.



> "No UserGroupInformation while generating ORC splits" - hive known issue in 
> 1.2.0-mapr-1607 release.
> 
>
> Key: DRILL-4874
> URL: https://issues.apache.org/jira/browse/DRILL-4874
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Tools, Build & Test
>Affects Versions: 1.7.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
> Fix For: 1.8.0
>
>
> Need upgrade drill to 1.2.0-mapr-1608 hive.version where [hive issue 
> HIVE-13120|https://issues.apache.org/jira/browse/HIVE-13120] is fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4874) "No UserGroupInformation while generating ORC splits" - hive known issue in 1.2.0-mapr-1607 release.

[Vitalii Diravka (JIRA)]

Vitalii Diravka created DRILL-4874:
--

 Summary: "No UserGroupInformation while generating ORC splits" - 
hive known issue in 1.2.0-mapr-1607 release.
 Key: DRILL-4874
 URL: https://issues.apache.org/jira/browse/DRILL-4874
 Project: Apache Drill
  Issue Type: Bug
  Components: Tools, Build & Test
Affects Versions: 1.7.0
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
 Fix For: 1.8.0


Need upgrade drill to 1.2.0-mapr-1608 hive.version where hvie issue 
[HIVE-13120|https://issues.apache.org/jira/browse/HIVE-13120] is fixed.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-4542) if external sort fails to spill to disk, memory is leaked and wrong error message is displayed

[Zelaine Fong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zelaine Fong updated DRILL-4542:

Assignee: Boaz Ben-Zvi

> if external sort fails to spill to disk, memory is leaked and wrong error 
> message is displayed
> --
>
> Key: DRILL-4542
> URL: https://issues.apache.org/jira/browse/DRILL-4542
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.6.0
> Environment: When external sort fails spilling to disk, for example 
> because there is no space left on disk, the query fails with an error like 
> this:
> {noformat}
> SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory 
> leaked: (376832)
> Allocator(op:1:0:6:ExternalSort) 2000/376832/408432256/429496729 
> (res/actual/peak/limit)
> {noformat}
> 2 problems here:
> 1. we are leaking memory
> 2. the "real" problem is not shown in the error message
> Looking at the logs we can find the real issue, generally right after the 
> previous error (edited for the sake of clarity):
> {noformat}
> 2016-03-26 02:22:28,576 [drill-executor-119] ERROR 
> o.a.d.exec.server.BootStrapContext - 
> org.apache.drill.exec.work.WorkManager$WorkerBee$2.run() leaked an exception.
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:249)
>  ~[hadoop-common-2.7.0-mapr-1602.jar:na]
>   [...]
> at 
> org.apache.drill.exec.cache.VectorAccessibleSerializable.writeToStream(VectorAccessibleSerializable.java:172)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.BatchGroup.addBatch(BatchGroup.java:96)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:562)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:394)
>  ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>   [...]
> {noformat}
>Reporter: Deneche A. Hakim
>Assignee: Boaz Ben-Zvi
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4873) Select from an array of empty maps results in NPE

[Khurram Faraaz (JIRA)]

Khurram Faraaz created DRILL-4873:
-

 Summary: Select from an array of empty maps results in NPE
 Key: DRILL-4873
 URL: https://issues.apache.org/jira/browse/DRILL-4873
 Project: Apache Drill
  Issue Type: Bug
  Components: Execution - Flow
Affects Versions: 1.9.0
Reporter: Khurram Faraaz


Select map from an array of empty maps results in NPE
Drill 1.9.0 git commit ID: 28d315bb

{noformat}
0: jdbc:drill:schema=dfs.tmp> select arr from `repeat_in_arry.json`;
+-+
|   arr   |
+-+
| [{},{},{},{},{},{},{}]  |
+-+
1 row selected (0.385 seconds)
{noformat}

Data used in test contains a single array of empty maps.
{noformat}
[root@centos-01 ~]# cat repeat_in_arry.json
{"arr":[{},{},{},{},{},{},{}]}
{noformat}

{noformat}
0: jdbc:drill:schema=dfs.tmp> select arr[1] from `repeat_in_arry.json`;
Error: Unexpected RuntimeException: java.lang.NullPointerException 
(state=,code=0)
{noformat}

Stack trace 
{noformat}
0: jdbc:drill:schema=dfs.tmp> select arr[1] from `repeat_in_arry.json`;
Error: Unexpected RuntimeException: java.lang.NullPointerException 
(state=,code=0)
java.sql.SQLException: Unexpected RuntimeException: 
java.lang.NullPointerException
at 
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:261)
at 
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1933)
at 
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:73)
at 
net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
at 
net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
at 
net.hydromatic.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:338)
at 
net.hydromatic.avatica.AvaticaStatement.execute(AvaticaStatement.java:69)
at 
org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101)
at sqlline.Commands.execute(Commands.java:841)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:746)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Caused by: java.lang.NullPointerException
at 
org.apache.drill.exec.record.RecordBatchLoader.load(RecordBatchLoader.java:117)
at 
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:223)
... 14 more
{noformat}  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459452#comment-15459452
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77403990
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java
 ---
@@ -141,25 +141,41 @@ public void validate(final OptionValue v, final 
OptionManager manager) {
 
   public static class BooleanValidator extends TypeValidator {
 public BooleanValidator(String name, boolean def) {
-  super(name, Kind.BOOLEAN, 
OptionValue.createBoolean(OptionType.SYSTEM, name, def));
+  this(name, def, false);
+}
+
+public BooleanValidator(String name, boolean def, boolean 
isAdminOption) {
+  super(name, Kind.BOOLEAN, 
OptionValue.createBoolean(OptionType.SYSTEM, name, def), isAdminOption);
 }
   }
 
   public static class StringValidator extends TypeValidator {
 public StringValidator(String name, String def) {
-  super(name, Kind.STRING, OptionValue.createString(OptionType.SYSTEM, 
name, def));
+  this(name, def, false);
+}
+
+public StringValidator(String name, String def, boolean isAdminOption) 
{
--- End diff --

See above comment.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459450#comment-15459450
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77403860
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java
 ---
@@ -141,25 +141,41 @@ public void validate(final OptionValue v, final 
OptionManager manager) {
 
   public static class BooleanValidator extends TypeValidator {
 public BooleanValidator(String name, boolean def) {
-  super(name, Kind.BOOLEAN, 
OptionValue.createBoolean(OptionType.SYSTEM, name, def));
+  this(name, def, false);
+}
+
+public BooleanValidator(String name, boolean def, boolean 
isAdminOption) {
--- End diff --

Must be auto-generated code... Perhaps change "isAdminOption" to just 
"option" as a Boolean validator would seem to want to work with any boolean 
option...


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459444#comment-15459444
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77403503
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java
 ---
@@ -69,6 +75,13 @@ public int getTtl() {
   }
 
   /**
+   * @return true is option can be set on system level only
--- End diff --

Explain a bit better. What is meant by "on system level only"? That this 
can be set only for the whole Drill system? That this is a system-level 
property that only the admin (not user) can set?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4280) Kerberos Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459440#comment-15459440
 ] 

ASF GitHub Bot commented on DRILL-4280:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/578#discussion_r77403393
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/client/AuthenticationUtil.java
 ---
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.client;
+
+import org.apache.drill.common.config.ConnectionParams;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.CommonConfigurationKeys;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import javax.security.auth.callback.Callback;
+import javax.security.auth.callback.CallbackHandler;
+import javax.security.auth.callback.NameCallback;
+import javax.security.auth.callback.PasswordCallback;
+import javax.security.auth.callback.UnsupportedCallbackException;
+import javax.security.sasl.Sasl;
+import javax.security.sasl.SaslClient;
+import javax.security.sasl.SaslException;
+import java.io.IOException;
+import java.lang.reflect.UndeclaredThrowableException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.security.PrivilegedExceptionAction;
+
+public final class AuthenticationUtil {
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(AuthenticationUtil.class);
+
+  private static final String PLAIN_NAME = "PLAIN";
+
+  private static final String KERBEROS_NAME = "GSSAPI";
+
+
+  public static String getMechanismFromParams(final ConnectionParams 
params) {
+if (params.getParam(ConnectionParams.AUTH_MECHANISM) != null) {
+  return params.getParam(ConnectionParams.AUTH_MECHANISM);
+}
+if (params.getParam(ConnectionParams.PASSWORD) != null) {
+  return PLAIN_NAME;
+}
+if (params.getParam(ConnectionParams.PRINCIPAL) != null ||
+(params.getParam(ConnectionParams.SERVICE_HOST) != null &&
+params.getParam(ConnectionParams.SERVICE_NAME) != null)) {
+  return KERBEROS_NAME;
+}
+return null;
+  }
+
+  public static SaslClient getPlainSaslClient(final String userName, final 
String password) throws SaslException {
+return Sasl.createSaslClient(new String[]{PLAIN_NAME}, null /* 
authorizationID */, null, null, null,
+new CallbackHandler() {
+  @Override
+  public void handle(final Callback[] callbacks) throws 
IOException, UnsupportedCallbackException {
+for (final Callback callback : callbacks) {
+  if (callback instanceof NameCallback) {
+NameCallback.class.cast(callback).setName(userName);
+continue;
+  }
+  if (callback instanceof PasswordCallback) {
+
PasswordCallback.class.cast(callback).setPassword(password.toCharArray());
+continue;
+  }
+  throw new UnsupportedCallbackException(callback);
+}
+  }
+});
+  }
+
+  public static String deriveKerberosName(final ConnectionParams params) {
+final String principal = params.getParam(ConnectionParams.PRINCIPAL);
+if (principal != null) {
+  return principal;
+}
+
+final StringBuilder principalBuilder = new StringBuilder();
+final String serviceNameProp = 
params.getParam(ConnectionParams.SERVICE_NAME);
+if (serviceNameProp != null) {
+  principalBuilder.append(serviceNameProp);
+} else {
+  principalBuilder.append(System.getProperty("drill.service.name", 
"drill"));
--- End diff --

Can this not be null?


> Kerberos Authentication
> ---
>
> Key: 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459436#comment-15459436
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77403174
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDropFunction.java
 ---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DropFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlDropFunction extends DrillSqlCall {
+
+  private final SqlNode jar;
--- End diff --

The "boilerplate" is the same between this class and SqlCreateFunction. Is 
it worth factoring that stuff out into a common base class?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459434#comment-15459434
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77403003
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDropFunction.java
 ---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DropFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlDropFunction extends DrillSqlCall {
+
+  private final SqlNode jar;
+
+  public static final SqlSpecialOperator OPERATOR = new 
SqlSpecialOperator("DROP_FUNCTION", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, SqlParserPos 
pos, SqlNode... operands) {
+  return new SqlDropFunction(pos, operands[0]);
+}
+  };
+
+  public SqlDropFunction(SqlParserPos pos, SqlNode jar) {
+super(pos);
+this.jar = jar;
+  }
+
+  @Override
+  public SqlOperator getOperator() {
+return OPERATOR;
+  }
+
+  @Override
+  public List getOperandList() {
+List opList = Lists.newArrayList();
--- End diff --

See comment above.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459430#comment-15459430
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77402886
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlCreateFunction.java
 ---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.CreateFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlCreateFunction extends DrillSqlCall {
+
+  private final SqlNode jar;
+
+  public static final SqlSpecialOperator OPERATOR = new 
SqlSpecialOperator("CREATE_FUNCTION", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, SqlParserPos 
pos, SqlNode... operands) {
+  return new SqlCreateFunction(pos, operands[0]);
+}
+  };
+
+  public SqlCreateFunction(SqlParserPos pos, SqlNode jar) {
+super(pos);
+this.jar = jar;
+  }
+
+  @Override
+  public SqlOperator getOperator() {
+return OPERATOR;
+  }
+
+  @Override
+  public List getOperandList() {
+List opList = Lists.newArrayList();
--- End diff --

Java code says that the newArrayList( ) technique is obsolete. With 
parameterized types, the preferred form is now:

List opList = new ArrayList<>( );


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459419#comment-15459419
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77402584
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -106,11 +105,19 @@
   String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS =
   "drill.exec.debug.return_error_for_failure_in_cancelled_fragments";
 
+  String CLIENT_SUPPORT_COMPLEX_TYPES = 
"drill.client.supports-complex-types";
 
+  /**
+   * Configuration properties connected with dynamic UDFs support
+   */
--- End diff --

Are these only for dynamic UDFs? Do we have settings for (non-dynamic) 
UDFs? If so, should we have

drill.exec.udf
   prop1
   dynamic
  prop1
  prop2


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459413#comment-15459413
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77402402
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -106,11 +105,19 @@
   String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS =
   "drill.exec.debug.return_error_for_failure_in_cancelled_fragments";
 
+  String CLIENT_SUPPORT_COMPLEX_TYPES = 
"drill.client.supports-complex-types";
 
+  /**
+   * Configuration properties connected with dynamic UDFs support
+   */
+  String UDF_RETRY_TIMES = "drill.exec.udf.retry.times";
--- End diff --

Since retry is not a group, just use retry-times (or retry_times).



> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4872) NPE from CTAS partitioned by a projected casted null

[Khurram Faraaz (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459393#comment-15459393
 ] 

Khurram Faraaz commented on DRILL-4872:
---

Is this issue related to or "similar" to DRILL-4026 ?

> NPE from CTAS partitioned by a projected casted null
> 
>
> Key: DRILL-4872
> URL: https://issues.apache.org/jira/browse/DRILL-4872
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Boaz Ben-Zvi
>  Labels: NPE
> Fix For: Future
>
>
> Extracted from DRILL-3898 : Running the same test case on a smaller table ( 
> store_sales.dat from TPCDS SF 1) has no space issues, but there is a Null 
> Pointer Exception from the projection:
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.drill.exec.expr.fn.impl.ByteFunctionHelpers.compare(ByteFunctionHelpers.java:100)
>  ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.doEval(ProjectorTemplate.java:49)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen1.projectRecords(ProjectorTemplate.java:62)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:199)
>  ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> A simplified version of the test case:
> 0: jdbc:drill:zk=local> create table dfs.tmp.ttt partition by ( x ) as select 
> case when columns[8] = '' then cast(null as varchar(10)) else cast(columns[8] 
> as varchar(10)) end as x FROM 
> dfs.`/Users/boazben-zvi/data/store_sales/store_sales.dat`;
> Error: SYSTEM ERROR: NullPointerException
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DRILL-3898) No space error during external sort does not cancel the query

[Boaz Ben-Zvi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boaz Ben-Zvi updated DRILL-3898:

Attachment: sqlline_3898.ver_1_8.log

Log from 1.8 on embedded, with limited spill space (note no NPE).


> No space error during external sort does not cancel the query
> -
>
> Key: DRILL-3898
> URL: https://issues.apache.org/jira/browse/DRILL-3898
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0, 1.8.0
>Reporter: Victoria Markman
>Assignee: Boaz Ben-Zvi
> Fix For: Future
>
> Attachments: drillbit.log, sqlline_3898.ver_1_8.log
>
>
> While verifying DRILL-3732 I ran into a new problem.
> I think drill somehow loses track of out of disk exception and does not 
> cancel rest of the query, which results in NPE:
> Reproduction is the same as in DRILL-3732:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_20(ss_item_sk, 
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) 
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . >  select 
> . . . . . . . . . . . . >  case when columns[2] = '' then cast(null as 
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . >  case when columns[3] = '' then cast(null as 
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . >  case when columns[4] = '' then cast(null as 
> varchar(100)) else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[5] = '' then cast(null as 
> varchar(100)) else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[0] = '' then cast(null as 
> varchar(100)) else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[8] = '' then cast(null as 
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . >  from 
> . . . . . . . . . . . . >   `store_sales.dat` ss 
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 1:16
> [Error Id: 0ae9338d-d04f-4b4a-93aa-a80d13cedb29 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> This exception in drillbit.log should have triggered query cancellation:
> {code}
> 2015-10-06 17:01:34,463 [WorkManager-2] ERROR 
> o.apache.drill.exec.work.WorkManager - 
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:226)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) 
> ~[na:1.7.0_71]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
> ~[na:1.7.0_71]
> at java.io.FilterOutputStream.close(FilterOutputStream.java:157) 
> ~[na:1.7.0_71]
> at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:400)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.drill.exec.physical.impl.xsort.BatchGroup.close(BatchGroup.java:152)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:44) 
> ~[drill-common-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:553)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:362)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:104)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:94)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> 

[jira] [Commented] (DRILL-3898) No space error during external sort does not cancel the query

[Boaz Ben-Zvi (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459314#comment-15459314
 ] 

Boaz Ben-Zvi commented on DRILL-3898:
-

   Testing further 1.8 in embedded - the NPE shows up when there is enough 
spill disk space (or no spill). however when the space is restricted, the "No 
space left on device" error comes up, but the NPE does not !!  And the table 
was not created. (log is attached).
Which implies that in the current code, the query cancellation does propagate 
correctly.
May need to test again on a cluster, with the full SF100 before closing this 
bug.

  

> No space error during external sort does not cancel the query
> -
>
> Key: DRILL-3898
> URL: https://issues.apache.org/jira/browse/DRILL-3898
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.2.0, 1.8.0
>Reporter: Victoria Markman
>Assignee: Boaz Ben-Zvi
> Fix For: Future
>
> Attachments: drillbit.log
>
>
> While verifying DRILL-3732 I ran into a new problem.
> I think drill somehow loses track of out of disk exception and does not 
> cancel rest of the query, which results in NPE:
> Reproduction is the same as in DRILL-3732:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_20(ss_item_sk, 
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) 
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . >  select 
> . . . . . . . . . . . . >  case when columns[2] = '' then cast(null as 
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . >  case when columns[3] = '' then cast(null as 
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . >  case when columns[4] = '' then cast(null as 
> varchar(100)) else cast(columns[4] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[5] = '' then cast(null as 
> varchar(100)) else cast(columns[5] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[0] = '' then cast(null as 
> varchar(100)) else cast(columns[0] as varchar(100)) end, 
> . . . . . . . . . . . . >  case when columns[8] = '' then cast(null as 
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . >  from 
> . . . . . . . . . . . . >   `store_sales.dat` ss 
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 1:16
> [Error Id: 0ae9338d-d04f-4b4a-93aa-a80d13cedb29 on atsqa4-133.qa.lab:31010] 
> (state=,code=0)
> {code}
> This exception in drillbit.log should have triggered query cancellation:
> {code}
> 2015-10-06 17:01:34,463 [WorkManager-2] ERROR 
> o.apache.drill.exec.work.WorkManager - 
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:226)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) 
> ~[na:1.7.0_71]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 
> ~[na:1.7.0_71]
> at java.io.FilterOutputStream.close(FilterOutputStream.java:157) 
> ~[na:1.7.0_71]
> at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:400)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>  ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) 
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at 
> org.apache.drill.exec.physical.impl.xsort.BatchGroup.close(BatchGroup.java:152)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:44) 
> ~[drill-common-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:553)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:362)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
> at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
>  ~[drill-java-exec-1.2.0.jar:1.2.0]
>

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459311#comment-15459311
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77396445
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -106,11 +105,19 @@
   String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS =
   "drill.exec.debug.return_error_for_failure_in_cancelled_fragments";
 
+  String CLIENT_SUPPORT_COMPLEX_TYPES = 
"drill.client.supports-complex-types";
 
+  /**
+   * Configuration properties connected with dynamic UDFs support
+   */
+  String UDF_RETRY_TIMES = "drill.exec.udf.retry.times";
+  String UDF_DIRECTORY_STAGING = "drill.exec.udf.directory.staging";
+  String UDF_DIRECTORY_REGISTRY = "drill.exec.udf.directory.registry";
+  String UDF_DIRECTORY_TMP = "drill.exec.udf.directory.tmp";
+  String UDF_DIRECTORY_BASE = "drill.exec.udf.directory.base";
+  String UDF_DIRECTORY_FS = "drill.exec.udf.directory.fs";
--- End diff --

Let's discuss. DoY uses DFS to store files. Let's define a general property 
drill.dfs.home that points to the root DFS home folder. This can default to 
"/user/drill" for HDFS. ("/user" is an HDFS standard.) Users can change the 
root in a single place. Then, when merging DoY, we'll use the new 
drill.dfs.home property in place of the one that DoY currently uses.

Note also that Typesafe allows property values of the form:

drill.exec.udf.directory.fs: "$drill.dfs.home/udf" to make it easy to 
compute derived paths.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459293#comment-15459293
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77395696
  
--- Diff: exec/java-exec/src/main/codegen/data/Parser.tdd ---
@@ -38,7 +38,8 @@
 "REFRESH",
 "METADATA",
 "DATABASE",
-"IF"
+"IF",
+"JAR"
--- End diff --

Should these be alphabetical?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459292#comment-15459292
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77395651
  
--- Diff: distribution/src/resources/sqlline.bat ---
@@ -114,6 +114,10 @@ if "test%DRILL_LOG_DIR%" == "test" (
   set DRILL_LOG_DIR=%DRILL_HOME%\log
 )
 
+if "test%DRILL_UDF_DIR%" == "test" (
+  set DRILL_UDF_DIR=%DRILL_HOME%\udf
+)
+
--- End diff --

%DRILL_CONF_DIR%\udf
See comment above.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459287#comment-15459287
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77395469
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then
   fatal_error "Java 1.7 or later is required to run Apache Drill."
 fi
 
+# Create Drill local udf area, if it does not exist
+DRILL_UDF_DIR="$DRILL_HOME/udf"
+if [[ ! -d "$DRILL_UDF_DIR" ]]; then
+  mkdir -p "$DRILL_UDF_DIR"
+fi
+
 # Adjust paths for CYGWIN
 if $is_cygwin; then
   DRILL_HOME=`cygpath -w "$DRILL_HOME"`
   DRILL_CONF_DIR=`cygpath -w "$DRILL_CONF_DIR"`
   DRILL_LOG_DIR=`cygpath -w "$DRILL_LOG_DIR"`
+  DRILL_UDF_DIR=`cygpath -w "$DRILL_UDF_DIR"`
--- End diff --

Do this only if a DRILL_UDF_DIR exists. Actually, not sure if this is even 
needed if we compute $DRILL_UDF_DIR as $DRILL_CONF_DIR/udf.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459284#comment-15459284
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77395331
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then
   fatal_error "Java 1.7 or later is required to run Apache Drill."
 fi
 
+# Create Drill local udf area, if it does not exist
+DRILL_UDF_DIR="$DRILL_HOME/udf"
+if [[ ! -d "$DRILL_UDF_DIR" ]]; then
+  mkdir -p "$DRILL_UDF_DIR"
+fi
+
--- End diff --

This should be $DRILL_CONF_DIR/udf, not $DRILL_HOME. $DRILL_CONF_DIR points 
to the user's "site" directory, which will be outside of $DRILL_HOME for 
Drill-on-YARN.

Not sure we need to create this directory. The pattern followed elsewhere 
is to include the directory on the class path if it exists.

In Drill-on-YARN, we CAN'T create the directory; the "localized" Drill and 
site directories are read-only.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-3178) csv reader should allow newlines inside quotes

[JIRA]

[ 
https://issues.apache.org/jira/browse/DRILL-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458600#comment-15458600
 ] 

F Méthot commented on DRILL-3178:
-

With 1.7 build, for this file:

> cat data/3428.csv
1,"line1"
2,"line2
"
3,"line3"

I get:

> select * from my_dfs.`/root/data/3428.csv`;
Error: DATA_READ ERROR: Error processing input: Cannot use newline character 
within quoted string, line=3, char=22. Content parsed: [ ]

Failure while reading file file:///root/data/3428.csv. Happened at or shortly 
before byte position 22.
Fragment 0:0

[Error Id: 49a05427-e763-4cca-9f97-e4b4308ecb75 on perfnode206.perf.lab:31010] 
(state=,code=0)



> csv reader should allow newlines inside quotes 
> ---
>
> Key: DRILL-3178
> URL: https://issues.apache.org/jira/browse/DRILL-3178
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Text & CSV
>Affects Versions: 1.0.0
> Environment: Ubuntu Trusty 14.04.2 LTS
>Reporter: Neal McBurnett
> Fix For: Future
>
>
> When reading a csv file which contains newlines within quoted strings, e.g. 
> via
> select * from dfs.`/tmp/q.csv`;
> Drill 1.0 says:
> Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException:  
> Error processing input: Cannot use newline character within quoted string
> But many tools produce csv files with newlines in quoted strings.  Drill 
> should be able to handle them.
> Workaround: the csvquote program (https://github.com/dbro/csvquote) can 
> encode embedded commas and newlines, and even decode them later if desired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)