[jira] [Created] (DRILL-4886) Merge maprdb format plugin source code

[Aditya Kishore (JIRA)]

Aditya Kishore created DRILL-4886:
-

 Summary: Merge maprdb format plugin source code
 Key: DRILL-4886
 URL: https://issues.apache.org/jira/browse/DRILL-4886
 Project: Apache Drill
  Issue Type: Task
  Components: Storage - Other
Reporter: Aditya Kishore
Assignee: Aditya Kishore
 Fix For: 1.9.0


Unlike other storage and format plugins, maprdb format plugin source code has 
so far lived in a private MapR repository which quite frequently causes the 
plugin to break due to fast-evolving nature of Drill interfaces.

Creating this JIRA to merge the maprdb format plugin source with Apache Drill 
source code.

Since the plugin depends upon a few non-opensource jars, the binary 
distribution convenience packages of Apache Drill will not include either the 
maprdb format-plugin or its dependencies.

Please see [LEGAL-251|https://issues.apache.org/jira/browse/LEGAL-251] for 
discussion/conclusion regarding inclusion of source code with non-open-source 
dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298816
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
+}
+
+// backup binary & source (copy to udf tmp directory)
+fs.mkdirs(tmpDir);
+Path tmpBinary = new Path(tmpDir, jarName);
+Path tmpSource = new 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299516
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
+}
+
+// backup binary & source (copy to udf tmp directory)
+fs.mkdirs(tmpDir);
+Path tmpBinary = new Path(tmpDir, jarName);
+Path tmpSource = new 

[GitHub] drill issue #586: DRILL-4886: Merge maprdb format plugin source code

[jacques-n]

Github user jacques-n commented on the issue:

https://github.com/apache/drill/pull/586
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #586: DRILL-4886: Merge maprdb format plugin source code

[adityakishore]

GitHub user adityakishore opened a pull request:

https://github.com/apache/drill/pull/586

DRILL-4886: Merge maprdb format plugin source code

These commits add MapR Format Plugin to Drill code base.

The default build/test/packaging behavior for mapr-format-plugin module are

1. BUILD of mapr-format-plugin is ENABLED.
2. Unit tests of mapr-format-plugin module are DISABLED (use `-Pmapr` to 
enable).
3. Packaging of mapr-format-plugin is DISABLED (use `-Pmapr` to enable).

Please see [LEGAL-251](https://issues.apache.org/jira/browse/LEGAL-251) for 
discussion/conclusion regarding inclusion of source code with non-open-source 
dependency.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adityakishore/drill maprdb-plugin-merge

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/586.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #586


commit cb6cc78b9991e9d6e384a2cc28b70332a9d92a22
Author: Patrick Wong 
Date:   2014-09-09T20:35:54Z

Initial commit

commit 95face3575817719a389fe049d3d8c1d65bb12dd
Author: Steven Phillips 
Date:   2014-09-23T18:45:27Z

First version of maprdb plugin

+ Moved files under contrib/format-maprdb

commit bfd27ec82851446a918b65f06805c63f4f0690ae
Author: Patrick Wong 
Date:   2014-10-08T15:20:11Z

Updated plugin version to 0.7.0-SNAPSHOT

commit 846451926a9f9c43b874ed0cdfb565186419bef9
Author: Anbang Xu 
Date:   2014-10-21T22:49:30Z

MD-12: Filters are not being pushed down through maprdb plugin

+ Reverted mapr-hbase back to MapR 4.0.1 release
+ Apply DRILL-1508 and DRILL-1567 to MapRDB plugin

1. 
https://github.com/apache/drill/commit/67df8cfa27041c9696255ee96e2d066e034f6f58
2. 
https://github.com/apache/drill/commit/71f5ad447823cba96cf73fb6709f8ed6cc0b63e9

commit 0e5b67e964946380fc33f278f0a20340ec338109
Author: Patrick Wong 
Date:   2014-12-03T20:03:29Z

Updated plugin version to 0.8.0-SNAPSHOT

commit 7e59a1ee16bbff98f636dc290eda6bb8785c0ab2
Author: Aditya Kishore 
Date:   2014-12-31T19:28:59Z

DRILL-1900: Fix numeric overflow problem in HBase stats calculation.

commit cb0a676dd542eaadba49776d22bdaf7b5b335720
Author: Aditya Kishore 
Date:   2015-01-28T10:21:26Z

MD-125: Update HBase storage plugin to support HBase 0.98

commit bed705556be5bd0ed55366620f116d93aaf2237e
Author: vkorukanti 
Date:   2015-02-06T15:18:06Z

MD-143: Update DrillFileSystem package location after DRILL-2080.

commit d5de0b5566370e84abbc9c8bbadb64b7e153
Author: Aditya Kishore 
Date:   2015-02-25T01:12:08Z

MD-131: Queries fail when using maprdb plugin

commit b64ee82837d4ccb5d33d509787507bf9704e0820
Author: Aditya Kishore 
Date:   2015-03-25T21:57:28Z

MD-192: CONVERT_FROM in where clause

* Set the writeIndex of ByteBuf returned by Unpooled.wrappedBuffer() to 0.

commit 1c3013af5053136b167dd948f87f88092d2aa9d9
Author: Patrick Wong 
Date:   2015-04-15T20:01:32Z

Updated plugin version to 0.9.0-SNAPSHOT

commit 419e7f38f78428605000fd12873f4aa6eb7211c7
Author: vkorukanti 
Date:   2015-04-22T00:51:37Z

MD-244: Update MapRDB plugin after DRILL-2514 and DRILL-2413

commit c71ec6760ace959146174489e9551666880deb69
Author: Aditya Kishore 
Date:   2015-04-22T18:02:15Z

MD-245: Optiq is now Calcite.

commit ec901a5f1412e9e119f243745b816532365d54fb
Author: Patrick Wong 
Date:   2015-05-04T19:08:12Z

Updated version to 1.0.0-SNAPSHOT

commit 1c43e3d2f23cf96027d0b2940c7a6362880a5781
Author: Aditya Kishore 
Date:   2015-05-06T00:28:37Z

DRILL-2826: Simplify and centralize Operator cleanup

commit d7fe7825ce833c37a320ab7b94b07f5440a5c69f
Author: Aditya Kishore 
Date:   2015-05-11T22:40:33Z

MD-272: Incorporate DRILL-2848 (option to disable decimal data type)

commit a19d0164a17952d44ece1e98b0f6192ecfb5b2e8
Author: Patrick Wong 
Date:   2015-05-20T21:52:48Z

Updated plugin version to 1.1.0-SNAPSHOT

+Using MapR v4.1.0 dependencies

commit c5c24792b83f71826aea61ec9870146de48cfe98
Author: Aditya Kishore 
Date:   2016-08-26T18:37:00Z

MD-290: Fix build failure due to upstream changes.

+ Includes MD-292

commit ba8a785c9c932f7e9e90a28f1b67042e50126597
Author: Patrick Wong 
Date:   2015-07-06T03:30:03Z

Updated version to 1.2.0-SNAPSHOT

commit df19a019117db3448a01f34daf804df747ca94b8
Author: Smidth Panchamia 
Date:   2015-08-06T22:50:44Z

[GitHub] drill issue #586: DRILL-4886: Merge maprdb format plugin source code

[Agirish]

Github user Agirish commented on the issue:

https://github.com/apache/drill/pull/586
  
+1
Functional and TPC-H SF100 tests from [1] successful. Also verified 
building Drill with mapr profile and accessing MapR-DB binary tables using the 
maprdb format plugin. 

[1] https://github.com/mapr/drill-test-framework


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298630
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
--- End diff --

Agree. This will be covered with inner class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298649
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
--- End diff --

Agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298593
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
--- End diff --

Agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298762
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
+}
+
+// backup binary & source (copy to udf tmp directory)
+fs.mkdirs(tmpDir);
+Path tmpBinary = new Path(tmpDir, jarName);
+Path tmpSource = new 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299671
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
+  repeated common.MajorType major_type = 2;
--- End diff --

We are going to get rid of MajorType and use String representation like 
VARCHAR-REQUIRED, INT-REQUIRED


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78300627
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testAbsentJarInStaging() throws Exception {
+String jar = "jar_name.jar";
+String staging = 
getDrillbitContext().getRemoteFunctionRegistry().getStagingArea().toUri().getPath();
+String summary = 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298889
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
+}
+
+// backup binary & source (copy to udf tmp directory)
+fs.mkdirs(tmpDir);
+Path tmpBinary = new Path(tmpDir, jarName);
+Path tmpSource = new 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78300858
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testAbsentJarInStaging() throws Exception {
+String jar = "jar_name.jar";
+String staging = 
getDrillbitContext().getRemoteFunctionRegistry().getStagingArea().toUri().getPath();
+String summary = 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298745
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 
remoteRegistry.getStagingArea().toUri().getPath()));
+}
+
+// backup binary & source (copy to udf tmp directory)
+fs.mkdirs(tmpDir);
+Path tmpBinary = new Path(tmpDir, jarName);
+Path tmpSource = new 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299260
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then
   fatal_error "Java 1.7 or later is required to run Apache Drill."
 fi
 
+# Create Drill local udf area, if it does not exist
+DRILL_UDF_DIR="$DRILL_HOME/udf"
+if [[ ! -d "$DRILL_UDF_DIR" ]]; then
+  mkdir -p "$DRILL_UDF_DIR"
+fi
+
 # Adjust paths for CYGWIN
 if $is_cygwin; then
   DRILL_HOME=`cygpath -w "$DRILL_HOME"`
   DRILL_CONF_DIR=`cygpath -w "$DRILL_CONF_DIR"`
   DRILL_LOG_DIR=`cygpath -w "$DRILL_LOG_DIR"`
+  DRILL_UDF_DIR=`cygpath -w "$DRILL_UDF_DIR"`
--- End diff --

Agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299594
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
   debug: {
 return_error_for_failure_in_cancelled_fragments: false
   }
+  udf: {
+retry.times: 5,
+directory: {
+  base: "/tmp",
--- End diff --

I'll change it to /user/drill


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78295758
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -69,29 +72,43 @@ public static PhysicalPlan getPlan(QueryContext 
context, String sql, Pointer

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298622
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
--- End diff --

Agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299325
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299537
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -0,0 +1,123 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlDropFunction;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.List;
+
+public class DropFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DropFunctionHandler.class);
+
+  public DropFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Drops UDFs dynamically.
+   * @return - Single row indicating list of unregistered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
--- End diff --

I'll add changes here as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294159
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294310
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294665
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294837
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -69,29 +72,43 @@ public static PhysicalPlan getPlan(QueryContext 
context, String sql, Pointerhttps://github.com/apache/calcite/blob/d9eb4383290e6ec7a5bca8d23fa3e03167c699fe/core/src/main/resources/org/apache/calcite/runtime/CalciteResource.properties#L41
[2] 
https://github.com/apache/calcite/blob/71df67c97f7b9a9d3fc6b92eec8ebdd1b7becfd1/core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java#L1575


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292645
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -138,18 +171,25 @@ private String functionReplacement(FunctionCall 
functionCall) {
 
   /**
* Find the Drill function implementation that matches the name, arg 
types and return type.
-   * @param name
-   * @param argTypes
-   * @param returnType
-   * @return
+   * If exact function implementation was not found and in case if Dynamic 
UDF Support is enabled
+   * loads all missing remote functions and tries to find Drill 
implementation one more time.
*/
   public DrillFuncHolder findExactMatchingDrillFunction(String name, 
List argTypes, MajorType returnType) {
+return findExactMatchingDrillFunction(name, argTypes, returnType, 
true);
+  }
+
+  private DrillFuncHolder findExactMatchingDrillFunction(String name, 
List argTypes, MajorType returnType, boolean retry) {
 for (DrillFuncHolder h : drillFuncRegistry.getMethods(name)) {
   if (h.matches(returnType, argTypes)) {
 return h;
   }
 }
 
+if (retry && optionManager != null && 
optionManager.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  if (loadRemoteFunctions()) {
--- End diff --

Agree. I'll make loadFunctions method thread-safe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292636
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
--- End diff --

It should be thread safe, I'll add DCL so two threads won't load functions 
at the same time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292785
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292766
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
--- End diff --

Agree. I'll add structure description.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292804
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = coordinator.getOrCreateTransientStore(jarsConfig);
+
+this.retryTimes = config.getInt(ExecConstants.UDF_RETRY_TIMES);
+
+Configuration conf = new Configuration();
+if 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293210
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+String sourceName = JarUtil.getSourceName(jarName);
+
+Path remoteBinary = new Path(registryArea, jarName);
+Path remoteSource = new Path(registryArea, sourceName);
+
+Path binary = new Path(localUdfArea, jarName);
+Path source = new Path(localUdfArea, sourceName);
+
+fs.copyToLocalFile(remoteBinary, binary);
+fs.copyToLocalFile(remoteSource, source);
--- End diff --

In this case, error is logged but I'll add better logging to show exactly 
on which step loading has failed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293364
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
--- End diff --

This class has two functions:
1. prepare environment for remote function registry which is done at start 
up (we have only one instance of this class per drillbit).
2. delegate actions to Zookeeper which handles synchronization by itself.
So we don't any synchronization in this class. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293473
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
--- End diff --

Transient store concept creation in Drill is the following:
when you connect to transient store you need to indicate the name (ex: 
name("udf/unregister")), Drill creates this path as persistent znode but 
anything else that you'll create under this znode will be transient.
I'll add some comments how it works.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78291908
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
+  Map map = jars.get(jName);
+
+  List list = map.get(fName);
+  if (list == null) {
+list = Lists.newArrayList();
+map.put(fName, list);
+  }
+
+  if (!list.contains(sName)) {
+list.add(sName);
+
+Map sigsMap = functions.get(fName);
+if (sigsMap == null) {
+  sigsMap = Maps.newHashMap();
+  functions.put(fName, sigsMap);
+}
+
+U u = sigsMap.get(sName);
+if (u == null) {
+  sigsMap.put(sName, fHolder);
+}
+  }
+}
+
+private void removeAllByJar(T jName) {
+  Map removeMap = jars.remove(jName);
+  if (removeMap == null) {
+return;
+  }
+
+  for (Map.Entry remEntry : removeMap.entrySet()) {
+Map fNameMap = 

[GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support

[arina-ielchiieva]

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292107
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
--- End diff --

Local UDF area must exists since we will create it in drill-config.sh, but 
I can add appropriate checks in getUdfDir method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---