[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588031#comment-15588031
 ] 

Arina Ielchiieva commented on DRILL-4726:
-

Merged into Apache master with commit 89f2633f612a645666de8f51dcb19c6f8044a95e

> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
>  Labels: doc-impacting
> Fix For: 1.9.0
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587059#comment-15587059
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/574


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566365#comment-15566365
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/574
  
+1 (non-binding)


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563402#comment-15563402
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82580650
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -324,6 +324,21 @@ if [ -n "$DRILL_CLASSPATH" ]; then
   CP="$CP:$DRILL_CLASSPATH"
 fi
 
+# If tmp dir is given, it must exist.
--- End diff --

Done.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563401#comment-15563401
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82618614
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -194,10 +194,12 @@ drill.exec: {
   udf: {
 retry-attempts: 5,
 directory: {
-  base: ${drill.home}"/"${drill.exec.zk.root}"/udf",
-  staging: ${drill.exec.udf.directory.base}"/staging",
-  registry: ${drill.exec.udf.directory.base}"/registry",
-  tmp: ${drill.exec.udf.directory.base}"/tmp"
+  base: ${drill.exec.zk.root}"/udf",
+  local: ${drill.exec.udf.directory.base}"/local",
--- End diff --

Done.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563406#comment-15563406
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82579439
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -324,6 +324,21 @@ if [ -n "$DRILL_CLASSPATH" ]; then
   CP="$CP:$DRILL_CLASSPATH"
 fi
 
+# If tmp dir is given, it must exist.
+if [ -n "$DRILL_TMP_DIR" ]; then
+  if [[ ! -d "$DRILL_TMP_DIR" ]]; then
+fatal_error "temporary dir does not exist:" $DRILL_TMP_DIR
--- End diff --

Done.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563403#comment-15563403
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82580748
  
--- Diff: distribution/src/resources/sqlline.bat ---
@@ -114,6 +114,10 @@ if "test%DRILL_LOG_DIR%" == "test" (
   set DRILL_LOG_DIR=%DRILL_HOME%\log
 )
 
+if "test%DRILL_TMP_DIR%" == "test" (
+  set DRILL_TMP_DIR=\tmp
--- End diff --

Done.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563404#comment-15563404
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82597440
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -378,38 +399,33 @@ private Path getLocalUdfDir() {
   }
 
   /**
-   * First tries to get drill conf directory value from system properties,
+   * First tries to get drill temporary directory value from system 
properties,
* if value is missing, checks environment properties.
* Throws exception is value is null.
-   * @return drill conf dir path
+   * @return drill temporary directory path
*/
-  private String getConfDir() {
-String drillConfDir = "DRILL_CONF_DIR";
-String value = System.getProperty(drillConfDir);
+  private String getTmpDir() {
--- End diff --

Done.
Added config ${drill.tmp-dir} which is not set by default.
So we first check if  environmental variable $DRILL_TMP_DIR is set, if not 
we check if ${drill.tmp-dir} is set.
If none of them is set we use Files.createTempDir( ) to generate temporary 
directory.
Generated directory will be deleted on drillbit exit. Directories indicated 
by $DRILL_TMP_DIR or ${drill.tmp-dir} will be only cleaned up 
($DRILL_TMP_DIR/udf or ${drill.tmp-dir}/udf).

Unfortunately I can set 
drill.tmp-dir: "/tmp",
drill.tmp-dir: "${?DRILL_TMP_DIR}" 
right away since typesafe config doesn't allow to unset value and thus user 
will never be able to have generated temporary directory.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563405#comment-15563405
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82597537
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -45,7 +45,7 @@ drill.client: {
   supports-complex-types: true
 }
 
-drill.home: "/tmp"
+drill.dfs-home: "/tmp"
--- End diff --

Decided to use new approach:
Introduced new config parameter ${drill.exec.udf.directory.root} which is 
not set by default.
User may override it if he wants to have custom root for remote udf areas.
If not set, fs.getHomeDirectory() is used.
For example (depending under which user drillbit is started):
Linux - /home/arina
Windows - /C:/Users/Arina
DFS - /user/root


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552886#comment-15552886
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82260717
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -45,7 +45,7 @@ drill.client: {
   supports-complex-types: true
 }
 
-drill.home: "/tmp"
+drill.dfs-home: "/tmp"
--- End diff --

On a standard HDFS setup, is there a /tmp folder? All the examples suggest 
that stuff often goes into "/user/something" such as "/user/drill".

Also, if "/tmp" behaves like a Linux /tmp, might something come along and 
clean up the UDFs during a long Drill run?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552883#comment-15552883
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82262083
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -194,10 +194,12 @@ drill.exec: {
   udf: {
 retry-attempts: 5,
 directory: {
-  base: ${drill.home}"/"${drill.exec.zk.root}"/udf",
-  staging: ${drill.exec.udf.directory.base}"/staging",
-  registry: ${drill.exec.udf.directory.base}"/registry",
-  tmp: ${drill.exec.udf.directory.base}"/tmp"
+  base: ${drill.exec.zk.root}"/udf",
+  local: ${drill.exec.udf.directory.base}"/local",
--- End diff --

How is this used? The path given by local seems relative:

drill/udf

Maybe include a simple comment to describe each config setting.

Otherwise, I like how this has come together!


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552881#comment-15552881
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82251309
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -324,6 +324,21 @@ if [ -n "$DRILL_CLASSPATH" ]; then
   CP="$CP:$DRILL_CLASSPATH"
 fi
 
+# If tmp dir is given, it must exist.
+if [ -n "$DRILL_TMP_DIR" ]; then
+  if [[ ! -d "$DRILL_TMP_DIR" ]]; then
+fatal_error "temporary dir does not exist:" $DRILL_TMP_DIR
--- End diff --

Capitalize "Temporary".


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552882#comment-15552882
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82252246
  
--- Diff: distribution/src/resources/sqlline.bat ---
@@ -114,6 +114,10 @@ if "test%DRILL_LOG_DIR%" == "test" (
   set DRILL_LOG_DIR=%DRILL_HOME%\log
 )
 
+if "test%DRILL_TMP_DIR%" == "test" (
+  set DRILL_TMP_DIR=\tmp
--- End diff --

On Windows, I think the standard is to use %TEMP%. There generally is no 
directory called "\tmp", sometimes there is a "\temp". But %TEMP% is the 
accepted practice.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552884#comment-15552884
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r82259602
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -378,38 +399,33 @@ private Path getLocalUdfDir() {
   }
 
   /**
-   * First tries to get drill conf directory value from system properties,
+   * First tries to get drill temporary directory value from system 
properties,
* if value is missing, checks environment properties.
* Throws exception is value is null.
-   * @return drill conf dir path
+   * @return drill temporary directory path
*/
-  private String getConfDir() {
-String drillConfDir = "DRILL_CONF_DIR";
-String value = System.getProperty(drillConfDir);
+  private String getTmpDir() {
--- End diff --

Can we be more forgiving here?

1. Use DRILL_TMP_DIR, if set.
2. Use a config file setting, if set.
3. Use Google's Files.createTempDir( ) which "Atomically creates a new 
directory somewhere beneath the system's temporary directory (as defined by the 
java.io.tmpdir system property)"

For most users, the choice in 3 should work fine. It would only be folks 
who have special needs that would set one of the other two properties.

Then, we can use a TypeSafe trick to combine 1 and 2. Define the config 
property something like this:

drill.temp-dir: "${DRILL_TMP_DIR"

Now, you just have to check drill.temp-dir in your function. If not set, 
use the Files approach as a default.

The nice thing about the Files approach is that each Drillbit will have a 
different directory (if two happen to be running (with different ports) at the 
same time.)

I wonder, however, does the Files temp directory get deleted on Drillbit 
exit?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552885#comment-15552885
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r8225
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -324,6 +324,21 @@ if [ -n "$DRILL_CLASSPATH" ]; then
   CP="$CP:$DRILL_CLASSPATH"
 fi
 
+# If tmp dir is given, it must exist.
--- End diff --

Please explain that DRILL_TMP_DIR is used for temporary storage of Dynamic 
UDF jars. (This comment helps folks understand that this is not the same as the 
tmp dir used for spill-to-disk...)


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526675#comment-15526675
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80648171
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
+Files.getNameWithoutExtension(jarName), System.nanoTime(), 
Files.getFileExtension(jarName));
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+Path remoteJar = new Path(registryArea, jarName);
+Path localJar = new Path(localUdfDir, generatedName);
+try {
+  fs.copyToLocalFile(remoteJar, localJar);
+} catch (IOException e) {
+  String message = String.format("Error during jar [%s] coping 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526686#comment-15526686
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80648485
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526677#comment-15526677
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80735634
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
--- End diff --

Unfortunately, I didn't realize that $DRILL_HOME and $DRILL_CONF_DIR in DoY 
are not writable.
Yes, you are right we do clean up local udf directory from previously 
loaded jars each time drillbit startups so it definitely a good idea to keep 
local udf directory in tmp folder,
So I have added $DRILL_TMP_DIR in drill-config.sh with default to /tmp if 
not set.
In code I concatenate $DRILL_TMP_DIR + drill.exec.udf.directory.base which 
defaults to ${drill.exec.zk.root}"/udf".



> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526680#comment-15526680
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80647672
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -179,14 +190,15 @@ public DrillFuncHolder 
findExactMatchingDrillFunction(String name, List argTypes, MajorType returnType, boolean retry) {
-for (DrillFuncHolder h : drillFuncRegistry.getMethods(name)) {
+AtomicLong version = new AtomicLong();
--- End diff --

We use AtomicLong to hold local registry version, when we call 
localFunctionRegistry.getMethods(name, version) we pass our AtomicLong and it's 
enriched with current local version registry number. Please see 
FunctionRegistryHolder.getMethods description for more details.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526683#comment-15526683
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80646286
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZookeeperClient.java
 ---
@@ -257,14 +263,47 @@ public void put(final String path, final byte[] data, 
DataChangeVersion version)
   }
   if (hasNode) {
 if (version != null) {
-  
curator.setData().withVersion(version.getVersion()).forPath(target, data);
+  try {
+
curator.setData().withVersion(version.getVersion()).forPath(target, data);
+  } catch (final KeeperException.BadVersionException e) {
+throw new VersionMismatchException("Unable to put data. 
Version mismatch is detected.", version.getVersion(), e);
+  }
 } else {
   curator.setData().forPath(target, data);
 }
   }
   getCache().rebuildNode(target);
-} catch (final KeeperException.BadVersionException e) {
-  throw new VersionMismatchException(e);
+} catch (final VersionMismatchException e) {
+  throw e;
+} catch (final Exception e) {
+  throw new DrillRuntimeException("unable to put ", e);
+}
+  }
+
+  /**
+   * Puts the given byte sequence into the given path if path is does not 
exist.
+   *
+   * @param path  target path
+   * @param data  data to store
+   * @return null if path was created, else data stored for the given path
+   */
+  public byte[] putIfAbsent(final String path, final byte[] data) {
+Preconditions.checkNotNull(path, "path is required");
+Preconditions.checkNotNull(data, "data is required");
+
+final String target = PathUtils.join(root, path);
+try {
+  boolean hasNode = hasPath(path, true);
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526676#comment-15526676
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80692935
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526678#comment-15526678
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80675326
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
+Files.getNameWithoutExtension(jarName), System.nanoTime(), 
Files.getFileExtension(jarName));
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+Path remoteJar = new Path(registryArea, jarName);
+Path localJar = new Path(localUdfDir, generatedName);
+try {
+  fs.copyToLocalFile(remoteJar, localJar);
+} catch (IOException e) {
+  String message = String.format("Error during jar [%s] coping 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526679#comment-15526679
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80669448
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
+Files.getNameWithoutExtension(jarName), System.nanoTime(), 
Files.getFileExtension(jarName));
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+Path remoteJar = new Path(registryArea, jarName);
+Path localJar = new Path(localUdfDir, generatedName);
+try {
+  fs.copyToLocalFile(remoteJar, localJar);
+} catch (IOException e) {
+  String message = String.format("Error during jar [%s] coping 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526685#comment-15526685
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80650009
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -48,54 +48,77 @@ public DropFunctionHandler(SqlHandlerConfig config) {
   }
 
   /**
-   * Drops UDFs dynamically.
+   * Unregisters UDFs dynamically. Process consists of several steps:
+   * 
+   * Registering jar in jar registry to ensure that several jars with 
the same name is not being unregistered.
+   * Starts remote unregistration process, gets list of all jars and 
excludes jar to be deleted.
+   * Signals drill bits to start local unregistration process.
+   * Removes source and binary jars from registry area.
+   * 
--- End diff --

As noted in previous comments, the expected behavior is to fail.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526684#comment-15526684
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80666252
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -120,9 +129,9 @@ public FunctionImplementationRegistry(DrillConfig 
config, ScanResult classpathSc
* Register functions in given operator table.
* @param operatorTable
*/
-  public void register(DrillOperatorTable operatorTable) {
+  public void register(DrillOperatorTable operatorTable, AtomicLong 
version) {
--- End diff --

Agree. It's used as holder to store local function registry version drill 
operator table was populated from.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526681#comment-15526681
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80649045
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526682#comment-15526682
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80649687
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526687#comment-15526687
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80737998
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -45,6 +45,8 @@ drill.client: {
   supports-complex-types: true
 }
 
+drill.home: "/tmp"
--- End diff --

Changed to drill.home to drill.dfs-home.
The problem why I set value to /tmp is since I am not quite sure to which 
value to set it.
If I set for example, /user/drill drillbit will fail at start up since 
currently we don't have such directory and user that runs drillbit usually 
doesn't have rights to create directory from /.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524281#comment-15524281
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80547641
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
--- End diff --

Unfortunately, this won't work in the case of Drill-on-YARN. The 
$DRILL_HOME and $DRILL_CONF_DIR directories are read-only in that case.

The new site directory (pointed to by DRILL_CONF_DIR) will contain a "jars" 
directory that contains statically-defined UDFs. In Drill-on-YARN, YARN copies 
all of the site directory to the local machine, but makes it read-only so that 
YARN can reuse that same "localized" copy for multiple runs. (That feature is 
handy fo map/reduce, but is not that useful for Drill. Still, that's how YARN 
works...)

One solution: provide a config option that specifies the local UDF 
location. The Apache Drill default can be the config dir (assuming there is a 
way to reference the config dir from within drill-override.conf -- need to 
check that.) For DoY, we will change the location to be a temp directory 
location provided by YARN.

Using the YARN temp directory ensures that the local udf dir starts out 
empty on each run. But, what about the "stock" Drill case? The 
$DRILL_CONFIG_DIR/udf directory probably will contain jars from a previous run. 
Is this desired? Does the code handle this case? Do we clean out UDFs that were 
dropped while the Drillbit was offline? Do we handle a partially-downloaded jar 
that was left incomplete when the previous run crashed?

Or, would it be better to clear the udf directory on the start of each 
Drill run? If we do that, can we always write udfs to a temp directory? Perhaps 
review the temp directories available.

Since DoY defines the temp directory at runtime, we need to set the temp 
diretory in drill-config.sh (which you did in a previous version.) As it turns 
out, Drill already has temp directories set in the config system (for 
spill-to-disk.) So we need to reconcile these two.

Perhaps this:

Define DRILL_TEMP_DIR in drill-config.sh. If it is set in the environment 
(the DoY case) or drill-env.sh (the non-DoY case), use it. Else, default to 
/tmp.

Under DoY, we can run multiple drillbits on the same host (by changing 
ports, etc.) So we need a unique path. Define the actual Drillbit temp 
directory to be

drillbit-temp-dir = $DRILL_TEMP_DIR/${drill-root}-${cluster-id}

We need both the root and cluster ID because neither is unique by itself, 
unfortunately.

Finally, udfs can reside in ${drillbit-temp-dir}/udf

This is just one possibility to illustrate the issue. Feel free to create a 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524288#comment-15524288
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80549839
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
+Files.getNameWithoutExtension(jarName), System.nanoTime(), 
Files.getFileExtension(jarName));
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+Path remoteJar = new Path(registryArea, jarName);
+Path localJar = new Path(localUdfDir, generatedName);
+try {
+  fs.copyToLocalFile(remoteJar, localJar);
+} catch (IOException e) {
+  String message = String.format("Error during jar [%s] coping from 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524283#comment-15524283
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80539844
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -120,9 +129,9 @@ public FunctionImplementationRegistry(DrillConfig 
config, ScanResult classpathSc
* Register functions in given operator table.
* @param operatorTable
*/
-  public void register(DrillOperatorTable operatorTable) {
+  public void register(DrillOperatorTable operatorTable, AtomicLong 
version) {
--- End diff --

Perhaps describe the semantics (purpose) of the atomic long. Is this for 
function versioning? If so, perhaps add a description of how this works to the 
Javadoc comment for this class so that the reader understands what's happening.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524285#comment-15524285
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80548125
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
--- End diff --

File.createTempFile ?


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524292#comment-15524292
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80559233
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link DrillFuncHolder} 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524282#comment-15524282
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80548740
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -301,29 +323,120 @@ private ScanResult scan(ClassLoader classLoader, 
Path path, URL[] urls) throws I
 return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
   }
 }
-throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+throw new JarValidationException(String.format("Marker file %s is 
missing in %s",
 CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
   }
 
-  private static String getUdfDir() {
-return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  /**
+   * Return list of jars that are missing in local function registry
+   * but present in remote function registry.
+   *
+   * @param remoteFunctionRegistry remote function registry
+   * @param localFunctionRegistry local function registry
+   * @return list of missing jars
+   */
+  private List getMissingJars(RemoteFunctionRegistry 
remoteFunctionRegistry,
+  LocalFunctionRegistry 
localFunctionRegistry) {
+List remoteJars = 
remoteFunctionRegistry.getRegistry().getJarList();
+List localJars = localFunctionRegistry.getAllJarNames();
+List missingJars = Lists.newArrayList();
+for (Jar jar : remoteJars) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+return missingJars;
+  }
+
+  /**
+   * Creates local udf directory, if it doesn't exist.
+   * Checks if local is a directory and if current application has write 
rights on it.
+   * Attempts to clean up local idf directory in case jars were left after 
previous drillbit run.
+   *
+   * @return path to local udf directory
+   */
+  private Path getLocalUdfDir() {
+String confDir = getConfDir();
+File udfDir = new File(confDir, "udf");
+String udfPath = udfDir.getPath();
+udfDir.mkdirs();
+Preconditions.checkState(udfDir.exists(), "Local udf directory [%s] 
must exist", udfPath);
+Preconditions.checkState(udfDir.isDirectory(), "Local udf directory 
[%s] must be a directory", udfPath);
+Preconditions.checkState(udfDir.canWrite(), "Local udf directory [%s] 
must be writable for application user", udfPath);
+try {
+  FileUtils.cleanDirectory(udfDir);
+} catch (IOException e) {
+  throw new DrillRuntimeException("Error during local udf directory 
clean up", e);
+}
+return new Path(udfDir.toURI());
+  }
+
+  /**
+   * First tries to get drill conf directory value from system properties,
+   * if value is missing, checks environment properties.
+   * Throws exception is value is null.
+   * @return drill conf dir path
+   */
+  private String getConfDir() {
+String drillConfDir = "DRILL_CONF_DIR";
+String value = System.getProperty(drillConfDir);
+if (value == null) {
+  value = Preconditions.checkNotNull(System.getenv(drillConfDir), "%s 
variable is not set", drillConfDir);
+}
+return value;
+  }
+
+  /**
+   * Copies jar from remote udf area to local udf area with numeric suffix,
+   * in order to achieve uniqueness for each locally copied jar.
+   * Ex: DrillUDF-1.0.jar -> DrillUDF-1.0_12200255588.jar
+   *
+   * @param jarName jar name to be copied
+   * @param remoteFunctionRegistry remote function registry
+   * @return local path to jar that was copied
+   * @throws IOException in case of problems during jar coping process
+   */
+  private Path copyJarToLocal(String jarName, RemoteFunctionRegistry 
remoteFunctionRegistry) throws IOException {
+String generatedName = String.format(generated_jar_name_pattern,
+Files.getNameWithoutExtension(jarName), System.nanoTime(), 
Files.getFileExtension(jarName));
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+Path remoteJar = new Path(registryArea, jarName);
+Path localJar = new Path(localUdfDir, generatedName);
+try {
+  fs.copyToLocalFile(remoteJar, localJar);
+} catch (IOException e) {
+  String message = String.format("Error during jar [%s] coping from 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524291#comment-15524291
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80578885
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -48,54 +48,77 @@ public DropFunctionHandler(SqlHandlerConfig config) {
   }
 
   /**
-   * Drops UDFs dynamically.
+   * Unregisters UDFs dynamically. Process consists of several steps:
+   * 
+   * Registering jar in jar registry to ensure that several jars with 
the same name is not being unregistered.
+   * Starts remote unregistration process, gets list of all jars and 
excludes jar to be deleted.
+   * Signals drill bits to start local unregistration process.
+   * Removes source and binary jars from registry area.
+   * 
--- End diff --

Can we explain the semantics regarding in-flight queries? If query Q is 
executing while we unregister UDF U, what happens?

Suppose user A submits query Q. The foreman for Q finds that Q uses UDF U. 
The rest of the prepare takes a while.

At the same time, user B unregisters U (right after the Foreman for Q 
verified that U exists.)

The Foreman for Q now sends fragments to other Drillbits.

Should Q find that U is available on those Drillbits? Or, is it OK for Q to 
fail because at least one Drillbit has started to unregister U?

Now, supppose that Drill queues are turned on and Q sits in the queue for a 
minute before executing. Should it still find that U is available on the 
various Drillbits?

This all may be worked out, the point here is just to explain the intended 
behavior.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524289#comment-15524289
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80558464
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link DrillFuncHolder} 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524287#comment-15524287
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80557682
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link DrillFuncHolder} 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524290#comment-15524290
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80559591
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
 ---
@@ -0,0 +1,360 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn.registry;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Queues;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+/**
+ * Function registry holder stores function implementations by jar name, 
function name.
+ * Contains two maps that hold data by jars and functions respectively.
+ * Jars map contains each jar as a key and map of all its functions with 
collection of function signatures as value.
+ * Functions map contains function name as key and map of its signatures 
and function holder as value.
+ * All maps and collections used are concurrent to guarantee memory 
consistency effects.
+ * Such structure is chosen to achieve maximum speed while retrieving data 
by jar or by function name,
+ * since we expect infrequent registry changes.
+ * Holder is designed to allow concurrent reads and single writes to keep 
data consistent.
+ * This is achieved by {@link ReadWriteLock} implementation usage.
+ * Holder has number version which changes every time new jars are added 
or removed. Initial version number is 0.
+ * Also version is used when user needs data from registry with version it 
is based on.
+ *
+ * Structure example:
+ *
+ * JARS
+ * built-in   -> upper  -> upper(VARCHAR-REQUIRED)
+ *-> lower  -> lower(VARCHAR-REQUIRED)
+ *
+ * First.jar  -> upper  -> upper(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * Second.jar -> lower  -> lower(VARCHAR-OPTIONAL)
+ *-> custom_upper   -> custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL)
+ *
+ * FUNCTIONS
+ * upper-> upper(VARCHAR-REQUIRED)-> function holder for 
upper(VARCHAR-REQUIRED)
+ *  -> upper(VARCHAR-OPTIONAL)-> function holder for 
upper(VARCHAR-OPTIONAL)
+ *
+ * lower-> lower(VARCHAR-REQUIRED)-> function holder for 
lower(VARCHAR-REQUIRED)
+ *  -> lower(VARCHAR-OPTIONAL)-> function holder for 
lower(VARCHAR-OPTIONAL)
+ *
+ * custom_upper -> custom_upper(VARCHAR-REQUIRED) -> function holder for 
custom_upper(VARCHAR-REQUIRED)
+ *  -> custom_upper(VARCHAR-OPTIONAL) -> function holder for 
custom_upper(VARCHAR-OPTIONAL)
+ *
+ * custom_lower -> custom_lower(VARCHAR-REQUIRED) -> function holder for 
custom_lower(VARCHAR-REQUIRED)
+ *  -> custom_lower(VARCHAR-OPTIONAL) -> function holder for 
custom_lower(VARCHAR-OPTIONAL)
+ *
+ * where
+ * First.jar is jar name represented by String
+ * upper is function name represented by String
+ * upper(VARCHAR-REQUIRED) is signature name represented by String which 
consist of function name, list of input parameters
+ * function holder for upper(VARCHAR-REQUIRED) is {@link DrillFuncHolder} 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524286#comment-15524286
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80542988
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -179,14 +190,15 @@ public DrillFuncHolder 
findExactMatchingDrillFunction(String name, List argTypes, MajorType returnType, boolean retry) {
-for (DrillFuncHolder h : drillFuncRegistry.getMethods(name)) {
+AtomicLong version = new AtomicLong();
--- End diff --

Not entirely clear how the version works. We create it here and intialze it 
to 0. Then in loadRemoteFunctions( ), we compare the local version (always 0) 
to the registry version (which will be zero only until, presumably, the first 
load.) Are we missing an initialization here to the current registry version? 
Else, please explain how this works.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524293#comment-15524293
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80579839
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -45,6 +45,8 @@ drill.client: {
   supports-complex-types: true
 }
 
+drill.home: "/tmp"
--- End diff --

Drill home is /tmp? Please explain...

From below, it seems that "drill.home" means the Drill home directory in 
DFS. (The problem is, we already use $DRILL_HOME to mean the location of the 
Drill install.)

Maybe change this to "drill.dfs.home" or "drill.dfs-home" to make it clear 
that this is for DFS.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524280#comment-15524280
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r80534600
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/coord/zk/ZookeeperClient.java
 ---
@@ -257,14 +263,47 @@ public void put(final String path, final byte[] data, 
DataChangeVersion version)
   }
   if (hasNode) {
 if (version != null) {
-  
curator.setData().withVersion(version.getVersion()).forPath(target, data);
+  try {
+
curator.setData().withVersion(version.getVersion()).forPath(target, data);
+  } catch (final KeeperException.BadVersionException e) {
+throw new VersionMismatchException("Unable to put data. 
Version mismatch is detected.", version.getVersion(), e);
+  }
 } else {
   curator.setData().forPath(target, data);
 }
   }
   getCache().rebuildNode(target);
-} catch (final KeeperException.BadVersionException e) {
-  throw new VersionMismatchException(e);
+} catch (final VersionMismatchException e) {
+  throw e;
+} catch (final Exception e) {
+  throw new DrillRuntimeException("unable to put ", e);
+}
+  }
+
+  /**
+   * Puts the given byte sequence into the given path if path is does not 
exist.
+   *
+   * @param path  target path
+   * @param data  data to store
+   * @return null if path was created, else data stored for the given path
+   */
+  public byte[] putIfAbsent(final String path, final byte[] data) {
+Preconditions.checkNotNull(path, "path is required");
+Preconditions.checkNotNull(data, "data is required");
+
+final String target = PathUtils.join(root, path);
+try {
+  boolean hasNode = hasPath(path, true);
--- End diff --

Isn't this a race condition? What if some other client creates the node 
between the check for hasPath and the create call below? The ZK documentation 
is sparse here, but my tests show that create( ) will fail if the node exists; 
you'll get a NodeExistsException, so no need to do the hasPath check.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496135#comment-15496135
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r79155363
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
--- End diff --

Agree, I have already removed creation from sh script to Drill.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486797#comment-15486797
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78527103
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486782#comment-15486782
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78525988
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -69,29 +72,43 @@ public static PhysicalPlan getPlan(QueryContext 
context, String sql, Pointer Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482275#comment-15482275
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78300977
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
--- End diff --

I'll add tests:
1. when two threads try to load the same jar;
2. remote registry version mismatch;
3. two threads are trying to load remote functions locally.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482267#comment-15482267
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78300858
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482254#comment-15482254
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78300627
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482175#comment-15482175
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299671
  
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
   NESTED_LOOP_JOIN = 35;
   AVRO_SUB_SCAN = 36;
 }
+
+message Func {
+  optional string name = 1;
+  repeated common.MajorType major_type = 2;
--- End diff --

We are going to get rid of MajorType and use String representation like 
VARCHAR-REQUIRED, INT-REQUIRED


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482163#comment-15482163
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299594
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
   debug: {
 return_error_for_failure_in_cancelled_fragments: false
   }
+  udf: {
+retry.times: 5,
+directory: {
+  base: "/tmp",
--- End diff --

I'll change it to /user/drill


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482157#comment-15482157
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299537
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -0,0 +1,123 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlDropFunction;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.List;
+
+public class DropFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DropFunctionHandler.class);
+
+  public DropFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Drops UDFs dynamically.
+   * @return - Single row indicating list of unregistered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
--- End diff --

I'll add changes here as well.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482154#comment-15482154
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299516
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482135#comment-15482135
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299325
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482130#comment-15482130
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78299260
  
--- Diff: distribution/src/resources/drill-config.sh ---
@@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then
   fatal_error "Java 1.7 or later is required to run Apache Drill."
 fi
 
+# Create Drill local udf area, if it does not exist
+DRILL_UDF_DIR="$DRILL_HOME/udf"
+if [[ ! -d "$DRILL_UDF_DIR" ]]; then
+  mkdir -p "$DRILL_UDF_DIR"
+fi
+
 # Adjust paths for CYGWIN
 if $is_cygwin; then
   DRILL_HOME=`cygpath -w "$DRILL_HOME"`
   DRILL_CONF_DIR=`cygpath -w "$DRILL_CONF_DIR"`
   DRILL_LOG_DIR=`cygpath -w "$DRILL_LOG_DIR"`
+  DRILL_UDF_DIR=`cygpath -w "$DRILL_UDF_DIR"`
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482104#comment-15482104
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298889
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482091#comment-15482091
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298816
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482086#comment-15482086
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298762
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482083#comment-15482083
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298745
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482060#comment-15482060
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298630
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
--- End diff --

Agree. This will be covered with inner class.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482062#comment-15482062
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298649
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
+
+  RemoteFunctionRegistry remoteRegistry = 
context.getRemoteFunctionRegistry();
+  FileSystem fs = remoteRegistry.getFs();
+  Path tmpDir = new Path(remoteRegistry.getTmpArea(), 
UUID.randomUUID().toString());
+  File localTmpDir = Files.createTempDir();
+
+  boolean inProgress = false;
+  try {
+final String action = remoteRegistry.addToJars(jarName, 
RemoteFunctionRegistry.Action.REGISTRATION);
+if (!(inProgress = action == null)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Jar with %s name is used. Action: %s", 
jarName, action));
+}
+
+// verify that binary and source exist
+Path remoteBinary = new Path(remoteRegistry.getStagingArea(), 
jarName);
+Path remoteSource = new Path(remoteRegistry.getStagingArea(), 
sourceName);
+if (!fs.exists(remoteBinary) || !fs.exists(remoteSource)) {
+  return DirectPlan.createDirectPlan(context, false,
+  String.format("Binary [%s] or source [%s] is absent in udf 
staging area [%s].", jarName, sourceName, 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482058#comment-15482058
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298622
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  SqlCreateFunction node = unwrap(sqlNode, SqlCreateFunction.class);
+  String jarName = ((SqlCharStringLiteral) node.getJar()).toValue();
+  String sourceName = JarUtil.getSourceName(jarName);
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482050#comment-15482050
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78298593
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.handlers;
+
+import com.google.common.collect.Lists;
+import com.google.common.io.Files;
+import org.apache.calcite.sql.SqlCharStringLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.exception.FunctionValidationException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.physical.PhysicalPlan;
+import org.apache.drill.exec.planner.sql.DirectPlan;
+import org.apache.drill.exec.planner.sql.parser.SqlCreateFunction;
+import org.apache.drill.exec.proto.UserBitShared.Func;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.drill.exec.work.foreman.ForemanSetupException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import java.util.UUID;
+
+public class CreateFunctionHandler extends DefaultSqlHandler {
+
+  private static org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(CreateFunctionHandler.class);
+
+  public CreateFunctionHandler(SqlHandlerConfig config) {
+super(config);
+  }
+
+  /**
+   * Creates UDFs dynamically.
+   *
+   * @return - Single row indicating list of registered UDFs, raise 
exception otherwise
+   */
+  @Override
+  public PhysicalPlan getPlan(SqlNode sqlNode) throws 
ForemanSetupException, IOException {
+if 
(context.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481801#comment-15481801
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78295758
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillSqlWorker.java
 ---
@@ -69,29 +72,43 @@ public static PhysicalPlan getPlan(QueryContext 
context, String sql, Pointer Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481717#comment-15481717
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294665
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481689#comment-15481689
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294310
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481678#comment-15481678
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78294159
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481644#comment-15481644
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293473
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
--- End diff --

Transient store concept creation in Drill is the following:
when you connect to transient store you need to indicate the name (ex: 
name("udf/unregister")), Drill creates this path as persistent znode but 
anything else that you'll create under this znode will be transient.
I'll add some comments how it works.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481636#comment-15481636
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293364
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
--- End diff --

This class has two functions:
1. prepare environment for remote function registry which is done at start 
up (we have only one instance of this class per drillbit).
2. delegate actions to Zookeeper which handles synchronization by itself.
So we don't any synchronization in this class. 


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481623#comment-15481623
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78293210
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+String sourceName = JarUtil.getSourceName(jarName);
+
+Path remoteBinary = new Path(registryArea, jarName);
+Path remoteSource = new Path(registryArea, sourceName);
+
+Path binary = new Path(localUdfArea, jarName);
+Path source = new Path(localUdfArea, sourceName);
+
+fs.copyToLocalFile(remoteBinary, binary);
+fs.copyToLocalFile(remoteSource, source);
--- End diff --

In this case, error is logged but I'll add better logging to show exactly 
on which step loading has failed.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481576#comment-15481576
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292804
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481572#comment-15481572
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292785
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
+public class RemoteFunctionRegistry implements AutoCloseable {
+
+  public static final String REGISTRY = "registry";
+
+  private static final ObjectMapper mapper = new 
ObjectMapper().enable(INDENT_OUTPUT);
+
+  private final TransientStoreListener unregistrationListener;
+  private int retryTimes;
+  private FileSystem fs;
+  private Path registryArea;
+  private Path stagingArea;
+  private Path tmpArea;
+
+  private PersistentStore registry;
+  private TransientStore unregistration;
+  private TransientStore jars;
+
+  public RemoteFunctionRegistry(TransientStoreListener 
unregistrationListener) {
+this.unregistrationListener = unregistrationListener;
+  }
+
+  public void init(DrillConfig config, PersistentStoreProvider 
storeProvider, ClusterCoordinator coordinator) {
+try {
+  PersistentStoreConfig registrationConfig = 
PersistentStoreConfig
+  .newProtoBuilder(SchemaUserBitShared.Registry.WRITE, 
SchemaUserBitShared.Registry.MERGE)
+  .name("udf")
+  .persist()
+  .build();
+  registry = storeProvider.getOrCreateStore(registrationConfig);
+  registry.putIfAbsent(REGISTRY, Registry.getDefaultInstance());
+} catch (StoreException e) {
+  throw new DrillRuntimeException("Failure while loading remote 
registry.", e);
+}
+
+TransientStoreConfig unregistrationConfig = 
TransientStoreConfig.
+newJacksonBuilder(mapper, 
String.class).name("udf/unregister").build();
+unregistration = 
coordinator.getOrCreateTransientStore(unregistrationConfig);
+unregistration.addListener(unregistrationListener);
+
+TransientStoreConfig jarsConfig = TransientStoreConfig.
+newJacksonBuilder(mapper, String.class).name("udf/jars").build();
+jars = 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481569#comment-15481569
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292766
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/RemoteFunctionRegistry.java
 ---
@@ -0,0 +1,185 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.FilenameUtils;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.coord.ClusterCoordinator;
+import org.apache.drill.exec.coord.store.TransientStore;
+import org.apache.drill.exec.coord.store.TransientStoreConfig;
+import org.apache.drill.exec.coord.store.TransientStoreListener;
+import org.apache.drill.exec.exception.StoreException;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.proto.SchemaUserBitShared;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreConfig;
+import org.apache.drill.exec.store.sys.PersistentStoreProvider;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+import static 
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+
+/** Is responsible for remote function registry management.
+ *  Creates all remote registry areas at startup, during init establishes 
connections with stores.
+ *  Provides tools to update remote registry and unregister functions, 
access remote registry areas.
+ */
--- End diff --

Agree. I'll add structure description.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481561#comment-15481561
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292645
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -138,18 +171,25 @@ private String functionReplacement(FunctionCall 
functionCall) {
 
   /**
* Find the Drill function implementation that matches the name, arg 
types and return type.
-   * @param name
-   * @param argTypes
-   * @param returnType
-   * @return
+   * If exact function implementation was not found and in case if Dynamic 
UDF Support is enabled
+   * loads all missing remote functions and tries to find Drill 
implementation one more time.
*/
   public DrillFuncHolder findExactMatchingDrillFunction(String name, 
List argTypes, MajorType returnType) {
+return findExactMatchingDrillFunction(name, argTypes, returnType, 
true);
+  }
+
+  private DrillFuncHolder findExactMatchingDrillFunction(String name, 
List argTypes, MajorType returnType, boolean retry) {
 for (DrillFuncHolder h : drillFuncRegistry.getMethods(name)) {
   if (h.matches(returnType, argTypes)) {
 return h;
   }
 }
 
+if (retry && optionManager != null && 
optionManager.getOption(ExecConstants.DYNAMIC_UDF_SUPPORT_ENABLED).bool_val) {
+  if (loadRemoteFunctions()) {
--- End diff --

Agree. I'll make loadFunctions method thread-safe.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481560#comment-15481560
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292636
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
--- End diff --

It should be thread safe, I'll add DCL so two threads won't load functions 
at the same time.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481520#comment-15481520
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78292107
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
--- End diff --

Local UDF area must exists since we will create it in drill-config.sh, but 
I can add appropriate checks in getUdfDir method.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481498#comment-15481498
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78291908
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
+  Map map = jars.get(jName);
+
+  List list = map.get(fName);
+  if (list == null) {
+list = Lists.newArrayList();
+map.put(fName, list);
+  }
+
+  if (!list.contains(sName)) {
+list.add(sName);
+
+Map sigsMap = functions.get(fName);
+if (sigsMap == null) {
+  sigsMap = Maps.newHashMap();
+  functions.put(fName, sigsMap);
+}
+
+U u = sigsMap.get(sName);
+if (u == null) {
+  sigsMap.put(sName, fHolder);
+}
+  }
+}
+
+private void removeAllByJar(T jName) {

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477037#comment-15477037
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78177586
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
+  Map map = jars.get(jName);
+
+  List list = map.get(fName);
+  if (list == null) {
+list = Lists.newArrayList();
+map.put(fName, list);
+  }
+
+  if (!list.contains(sName)) {
+list.add(sName);
+
+Map sigsMap = functions.get(fName);
+if (sigsMap == null) {
+  sigsMap = Maps.newHashMap();
+  functions.put(fName, sigsMap);
+}
+
+U u = sigsMap.get(sName);
--- End diff --

You are right. Actually we don't expect any duplicates in jar, since we are 
adding jars only after validation.
I'll remove 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477036#comment-15477036
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78177558
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
+  Map map = jars.get(jName);
+
+  List list = map.get(fName);
+  if (list == null) {
+list = Lists.newArrayList();
+map.put(fName, list);
+  }
+
+  if (!list.contains(sName)) {
--- End diff --

You are right. Actually we don't expect any duplicates in jar, since we are 
adding jars only after validation.
I'll remove unnecessary checks.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>  

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477034#comment-15477034
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78177330
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
+  Map map = jars.get(jName);
+
+  List list = map.get(fName);
+  if (list == null) {
+list = Lists.newArrayList();
+map.put(fName, list);
+  }
+
+  if (!list.contains(sName)) {
+list.add(sName);
+
+Map sigsMap = functions.get(fName);
--- End diff --

Yes, signature includes function name: MY_FUNC(VARCHAR-REQUIRED, 
INT-REQUIRED)
But I'll add examples in description so it would be clearer.
We don't support name spaces in Drill, so we are save here.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476998#comment-15476998
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78174691
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
--- End diff --

It's used during local validation only. Since we have remote registry in 
Zookeeper and such race condition happens, we'll fail during remote validation 
anyway.



> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476977#comment-15476977
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78172686
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
--- End diff --

Agree. I suggest to go with geFunctionNamesByJar


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476975#comment-15476975
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78172574
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476925#comment-15476925
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78169406
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474290#comment-15474290
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78038409
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
--- End diff --

Agree.
1. I'll remove generics, it's overhead in this situation.
2. I make this class separate. Thus it will be public, so I'll be able to 
cover it with unit tests.



> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474182#comment-15474182
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78031041
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
--- End diff --

Agree. I'll add description with example of structure.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474180#comment-15474180
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78030875
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -64,62 +76,134 @@
   .put("CONVERT_FROM", Pair.of(2, 2))
   .put("FLATTEN", Pair.of(1, 1)).build();
 
+  /** Registers all functions present in Drill classpath on start-up. All 
functions will be marked as built-in.*/
   public DrillFunctionRegistry(ScanResult classpathScan) {
+validate(BUILT_IN, classpathScan);
+register(BUILT_IN, classpathScan, this.getClass().getClassLoader());
+if (logger.isTraceEnabled()) {
+  StringBuilder allFunctions = new StringBuilder();
+  for (DrillFuncHolder method: 
registryHolder.getAllFunctionsWithHolders().values()) {
+allFunctions.append(method.toString()).append("\n");
+  }
+  logger.trace("Registered functions: [\n{}]", allFunctions);
+}
+  }
+
+  /**
+   * Validates all functions, present in jars.
+   * Will throw {@link FunctionValidationException} if:
+   * 1. Jar with the same name has been already registered.
+   * 2. Conflicting function with the similar signature is found.
+   * 3. Aggregating function is not deterministic.
+   *
+   * @return list of validated functions
+   */
+  public List validate(String jarName, ScanResult classpathScan) {
+List functions = Lists.newArrayList();
 FunctionConverter converter = new FunctionConverter();
 List providerClasses = 
classpathScan.getAnnotatedClasses();
 
-// Hash map to prevent registering functions with exactly matching 
signatures
-// key: Function Name + Input's Major Type
-// value: Class name where function is implemented
-//
-final Map functionSignatureMap = new HashMap<>();
+if (registryHolder.containsJar(jarName)) {
+  throw new FunctionValidationException(String.format("Jar %s is 
already registered", jarName));
+}
+
+final ListMultimap allFuncWithSignatures = 
registryHolder.getAllFunctionsWithSignatures();
+
 for (AnnotatedClassDescriptor func : providerClasses) {
-  DrillFuncHolder holder = converter.getHolder(func);
+  DrillFuncHolder holder = converter.getHolder(func, 
ClassLoader.getSystemClassLoader());
   if (holder != null) {
-// register handle for each name the function can be referred to
-String[] names = holder.getRegisteredNames();
 
-// Create the string for input types
 String functionInput = "";
+List types = Lists.newArrayList();
 for (DrillFuncHolder.ValueReference ref : holder.parameters) {
   functionInput += ref.getType().toString();
+  types.add(ref.getType());
 }
+
+String[] names = holder.getRegisteredNames();
 for (String name : names) {
   String functionName = name.toLowerCase();
-  registeredFunctions.put(functionName, holder);
   String functionSignature = functionName + functionInput;
-  String existingImplementation;
-  if ((existingImplementation = 
functionSignatureMap.get(functionSignature)) != null) {
-throw new AssertionError(
-String.format(
-"Conflicting functions with similar signature found. 
Func Name: %s, Class name: %s " +
-" Class name: %s", functionName, func.getClassName(), 
existingImplementation));
-  } else if (holder.isAggregating() && !holder.isDeterministic() ) 
{
-logger.warn("Aggregate functions must be deterministic, did 
not register function {}", func.getClassName());
+
+  if 
(allFuncWithSignatures.get(functionName).contains(functionSignature)) {
--- End diff --

Explained in the comment above.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474179#comment-15474179
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78030815
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -64,62 +76,134 @@
   .put("CONVERT_FROM", Pair.of(2, 2))
   .put("FLATTEN", Pair.of(1, 1)).build();
 
+  /** Registers all functions present in Drill classpath on start-up. All 
functions will be marked as built-in.*/
   public DrillFunctionRegistry(ScanResult classpathScan) {
+validate(BUILT_IN, classpathScan);
+register(BUILT_IN, classpathScan, this.getClass().getClassLoader());
+if (logger.isTraceEnabled()) {
+  StringBuilder allFunctions = new StringBuilder();
+  for (DrillFuncHolder method: 
registryHolder.getAllFunctionsWithHolders().values()) {
+allFunctions.append(method.toString()).append("\n");
+  }
+  logger.trace("Registered functions: [\n{}]", allFunctions);
+}
+  }
+
+  /**
+   * Validates all functions, present in jars.
+   * Will throw {@link FunctionValidationException} if:
+   * 1. Jar with the same name has been already registered.
+   * 2. Conflicting function with the similar signature is found.
+   * 3. Aggregating function is not deterministic.
+   *
+   * @return list of validated functions
+   */
+  public List validate(String jarName, ScanResult classpathScan) {
+List functions = Lists.newArrayList();
 FunctionConverter converter = new FunctionConverter();
 List providerClasses = 
classpathScan.getAnnotatedClasses();
 
-// Hash map to prevent registering functions with exactly matching 
signatures
-// key: Function Name + Input's Major Type
-// value: Class name where function is implemented
-//
-final Map functionSignatureMap = new HashMap<>();
+if (registryHolder.containsJar(jarName)) {
+  throw new FunctionValidationException(String.format("Jar %s is 
already registered", jarName));
+}
--- End diff --

As you noted, built-in functions creation is save here, since they are 
registered at start up.

The race condition you are talking about is handled by remote registry 
versioning (thus by Zoookeeper itself).
As you know that we have two validation steps: local and remote.
So this method is responsible for local validation. 
Let's say we have:
Thread1 that registers Jar1 where F1(VARCHAR-REQUIRED) is present
Thread2 that registers Jar2 where F1(VARCHAR-REQUIRED) is present
Since F1(VARCHAR-REQUIRED) is absent in LOCAL function registry, both 
threads pass local validation successfully.
Then they start remote validation.
Each thread retrieves remote function registry with version 1.
Since F1(VARCHAR-REQUIRED) is absent in REMOTE function registry, both 
threads pass remote validation successfully.
Then each thread updates remote function registry and tries to send it to 
Zookeeper.
This part is controlled by Zookeeper, eventually one thread will send 
updated remote registry in Zookeeper first. and remote registry version will 
change to 2. So the other thread will get VersionMismatchException. In this 
case such thread will load remote registry with version 2 and execute remote 
validation again during which it will detect duplicates and send appropriate 
response to the user.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474152#comment-15474152
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78028875
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill;
+
+import mockit.NonStrictExpectations;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.common.util.TestTools;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.expr.fn.RemoteFunctionRegistry;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.util.JarUtil;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+public class TestDynamicUDFSupport extends BaseTestQuery {
+
+  private static final File jars = new File(TestTools.getWorkingPath() + 
"/src/test/resources/jars");
+  private static final String jName = "DrillUDF-1.0.jar";
+  private static String sName = JarUtil.getSourceName(jName);
+
+  @Rule
+  public final TemporaryFolder drillUdfDir = new TemporaryFolder();
+
+  @Rule
+  public final TemporaryFolder base = new TemporaryFolder();
+
+  @Before
+  public void setEnvVariables()
+  {
+new NonStrictExpectations(System.class)
+{
+  {
+invoke(System.class, "getenv", "DRILL_UDF_DIR");
+returns(drillUdfDir.getRoot().getPath());
+  }
+};
+  }
+
+  @Before
+  public void setup() {
+String root = base.getRoot().getPath();
+if 
(!root.equals(getDrillbitContext().getConfig().getString(ExecConstants.UDF_DIRECTORY_STAGING)))
 {
+Properties overrideProps = new Properties();
+  overrideProps.setProperty(ExecConstants.UDF_DIRECTORY_BASE, root);
+  updateTestCluster(1, DrillConfig.create(overrideProps));
+}
+  }
+
+  @Test
+  public void testSyntax() throws Exception {
+test("create function using jar 'jar_name.jar'");
+test("drop function using jar 'jar_name.jar'");
+  }
+
+  @Test
+  public void testEnableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = true");
+  test("create function using jar 'jar_name.jar'");
+  test("drop function using jar 'jar_name.jar'");
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public void testDisableDynamicSupport() throws Exception {
+try {
+  test("alter system set `exec.udf.enable_dynamic_support` = false");
+  String[] actions = new String[] {"create", "drop"};
+  String query = "%s function using jar 'jar_name.jar'";
+  for (String action : actions) {
+try {
+  test(String.format(query, action));
+} catch (UserRemoteException e) {
+  assertThat(e.getMessage(), containsString("Dynamic UDFs support 
is disabled."));
+}
+  }
+} finally {
+  test("alter system reset `exec.udf.enable_dynamic_support`");
+}
+  }
+
+  @Test
+  public 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474130#comment-15474130
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78027749
  
--- Diff: 
common/src/main/java/org/apache/drill/common/config/DrillConfig.java ---
@@ -164,6 +164,15 @@ public static DrillConfig create(String 
overrideFileResourcePathname, boolean en
   }
 
   /**
+   * Creates a drill configuration using the provided config file.
--- End diff --

We pass Config object which contains URL to drill-module.conf in our jar. 
drill-module.conf contains packages that allowed to be scanned. Based on this 
URL, drill config is created. During creation drill finds all packages listed 
in drill-module.conf and adds them in drill config.
Example of created drill config:
{
# 
jar:file:/home/osboxes/git_repo/drill/exec/java-exec/target/1473346938237-0/DrillUDF-1.0.jar!/drill-module.conf:
 1
"drill" : {
# 
jar:file:/home/osboxes/git_repo/drill/exec/java-exec/target/1473346938237-0/DrillUDF-1.0.jar!/drill-module.conf:
 1
"classpath" : {
# 
jar:file:/home/osboxes/git_repo/drill/exec/java-exec/target/1473346938237-0/DrillUDF-1.0.jar!/drill-module.conf:
 1
"scanning" : {
# 
jar:file:/home/osboxes/git_repo/drill/exec/java-exec/target/1473346938237-0/DrillUDF-1.0.jar!/drill-module.conf:
 1
"packages" : [
# 
jar:file:/home/osboxes/git_repo/drill/exec/java-exec/target/1473346938237-0/DrillUDF-1.0.jar!/drill-module.conf:
 1
"com.drill.udf"
]
}
}
}
}


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474026#comment-15474026
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78018512
  
--- Diff: 
common/src/main/java/org/apache/drill/common/scanner/RunTimeScan.java ---
@@ -75,4 +77,20 @@ public static ScanResult fromPrescan(DrillConfig config) 
{
 }
   }
 
+  /**
+   * Scanning packages at runtime only for marked paths and list of 
pre-scanned annotations
+   * @param config to retrieve the packages to scan
+   * @param markedPath list of paths where to scan
+   * @return the scan result
+   */
+  public static ScanResult dynamicPackageScan(DrillConfig config, Set 
markedPath) {
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474011#comment-15474011
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78017223
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillOperatorTable.java
 ---
@@ -64,6 +64,15 @@ public DrillOperatorTable(FunctionImplementationRegistry 
registry, OptionManager
 this.systemOptionManager = systemOptionManager;
   }
 
+  /** Cleanups all operator holders and reloads operators */
--- End diff --

Agree, thanks for noticing.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473992#comment-15473992
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78015486
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionInitializer.java
 ---
@@ -42,22 +42,29 @@
 public class FunctionInitializer {
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FunctionInitializer.class);
 
-  private String className;
-
+  private final String className;
+  private final ClassLoader classloader;
   private Map functionUnits = Maps.newHashMap();
   private Map methods;
   private List imports;
   private volatile boolean ready;
 
   /**
* @param className the fully qualified name of the class implementing 
the function
+   * @param classLoader system or custom class loader
--- End diff --

Sure.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473979#comment-15473979
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78014300
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+String sourceName = JarUtil.getSourceName(jarName);
+
+Path remoteBinary = new Path(registryArea, jarName);
+Path remoteSource = new Path(registryArea, sourceName);
+
+Path binary = new Path(localUdfArea, jarName);
+Path source = new Path(localUdfArea, sourceName);
+
+fs.copyToLocalFile(remoteBinary, binary);
+fs.copyToLocalFile(remoteSource, source);
+
+URL[] urls = {binary.toUri().toURL(), source.toUri().toURL()};
+ClassLoader classLoader = new URLClassLoader(urls);
+ScanResult scanResult = scan(classLoader, binary, urls);
+drillFuncRegistry.register(jarName, scanResult, classLoader);
+  } catch (IOException | FunctionValidationException e) {
+logger.error("Problem during remote functions load from {}", 
jarName, e);
+  }
+}
+
+return missingJars.size() > 0;
+  }
+
+  private ScanResult scan(ClassLoader classLoader, Path path, URL[] urls) 
throws IOException {
+Enumeration e = 
classLoader.getResources(CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME);
+while (e.hasMoreElements()) {
+  URL res = e.nextElement();
+  if (res.getPath().contains(path.toUri().getPath())) {
+DrillConfig drillConfig = 
DrillConfig.create(ConfigFactory.parseURL(res));
+return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
+  }
+}
+throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
+  }
+
+  private static String getUdfDir() {
+return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  }
+
+  /**
+   * Fires when jar name is submitted for unregistration.
+   * Will unregister all functions associated with the jar name
+   * and delete binary and source associated with the jar from local 
DRILL_UDF_DIR.
+   */
+  public class UnregistrationListener implements TransientStoreListener {
+
+@Override
+public void onChange(TransientStoreEvent event) {
+String jarName = (String) event.getValue();
+String sourceName = JarUtil.getSourceName(jarName);
+String localDir = getUdfDir();
+FileUtils.deleteQuietly(new File(localDir, jarName));
--- End diff --

On Linux you can delete file if someone is using it,

we don't allow classloaders to 

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473956#comment-15473956
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78012385
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+String sourceName = JarUtil.getSourceName(jarName);
+
+Path remoteBinary = new Path(registryArea, jarName);
+Path remoteSource = new Path(registryArea, sourceName);
+
+Path binary = new Path(localUdfArea, jarName);
+Path source = new Path(localUdfArea, sourceName);
+
+fs.copyToLocalFile(remoteBinary, binary);
+fs.copyToLocalFile(remoteSource, source);
+
+URL[] urls = {binary.toUri().toURL(), source.toUri().toURL()};
+ClassLoader classLoader = new URLClassLoader(urls);
+ScanResult scanResult = scan(classLoader, binary, urls);
+drillFuncRegistry.register(jarName, scanResult, classLoader);
+  } catch (IOException | FunctionValidationException e) {
+logger.error("Problem during remote functions load from {}", 
jarName, e);
+  }
+}
+
+return missingJars.size() > 0;
+  }
+
+  private ScanResult scan(ClassLoader classLoader, Path path, URL[] urls) 
throws IOException {
+Enumeration e = 
classLoader.getResources(CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME);
+while (e.hasMoreElements()) {
+  URL res = e.nextElement();
+  if (res.getPath().contains(path.toUri().getPath())) {
+DrillConfig drillConfig = 
DrillConfig.create(ConfigFactory.parseURL(res));
+return RunTimeScan.dynamicPackageScan(drillConfig, 
Sets.newHashSet(urls));
+  }
+}
+throw new FunctionValidationException(String.format("Marker file %s is 
missing in %s.",
+CommonConstants.DRILL_JAR_MARKER_FILE_RESOURCE_PATHNAME, 
path.getName()));
+  }
+
+  private static String getUdfDir() {
+return Preconditions.checkNotNull(System.getenv("DRILL_UDF_DIR"), 
"DRILL_UDF_DIR variable is not set");
+  }
+
+  /**
+   * Fires when jar name is submitted for unregistration.
+   * Will unregister all functions associated with the jar name
+   * and delete binary and source associated with the jar from local 
DRILL_UDF_DIR.
+   */
+  public class UnregistrationListener implements TransientStoreListener {
+
+@Override
+public void onChange(TransientStoreEvent event) {
+String jarName = (String) event.getValue();
+String sourceName = JarUtil.getSourceName(jarName);
+String localDir = getUdfDir();
+FileUtils.deleteQuietly(new File(localDir, jarName));
+FileUtils.deleteQuietly(new File(localDir, sourceName));
+drillFuncRegistry.unregister(jarName);

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473941#comment-15473941
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78011083
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
+   *
+   * @return true if at least functions from one jar were loaded
+   */
+  public boolean loadRemoteFunctions() {
+List missingJars = Lists.newArrayList();
+Registry registry = remoteFunctionRegistry.getRegistry();
+
+List localJars = drillFuncRegistry.getAllJarNames();
+for (Jar jar : registry.getJarList()) {
+  if (!localJars.contains(jar.getName())) {
+missingJars.add(jar.getName());
+  }
+}
+
+for (String jarName : missingJars) {
+  try {
+Path localUdfArea = new Path(new File(getUdfDir()).toURI());
+Path registryArea = remoteFunctionRegistry.getRegistryArea();
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+String sourceName = JarUtil.getSourceName(jarName);
+
+Path remoteBinary = new Path(registryArea, jarName);
+Path remoteSource = new Path(registryArea, sourceName);
+
+Path binary = new Path(localUdfArea, jarName);
+Path source = new Path(localUdfArea, sourceName);
+
+fs.copyToLocalFile(remoteBinary, binary);
+fs.copyToLocalFile(remoteSource, source);
+
+URL[] urls = {binary.toUri().toURL(), source.toUri().toURL()};
+ClassLoader classLoader = new URLClassLoader(urls);
+ScanResult scanResult = scan(classLoader, binary, urls);
+drillFuncRegistry.register(jarName, scanResult, classLoader);
+  } catch (IOException | FunctionValidationException e) {
+logger.error("Problem during remote functions load from {}", 
jarName, e);
+  }
+}
+
+return missingJars.size() > 0;
+  }
+
+  private ScanResult scan(ClassLoader classLoader, Path path, URL[] urls) 
throws IOException {
--- End diff --

First it scans jar for it's marker file which indicates with packages to 
scan, and if it exists returns scan result which contains list of classes, 
annotations, packages found in our jar.
Agree, I'll should have added description to method.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473913#comment-15473913
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78009045
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -186,4 +226,105 @@ public boolean isFunctionComplexOutput(String name) {
 return false;
   }
 
+  public RemoteFunctionRegistry getRemoteFunctionRegistry() {
+return remoteFunctionRegistry;
+  }
+
+  public List validate(Path path) throws IOException {
+URL url = path.toUri().toURL();
+URL[] urls = {url};
+ClassLoader classLoader = new URLClassLoader(urls);
+return drillFuncRegistry.validate(path.getName(), scan(classLoader, 
path, urls));
+  }
+
+  public void register(String jarName, ScanResult classpathScan, 
ClassLoader classLoader) {
+drillFuncRegistry.register(jarName, classpathScan, classLoader);
+  }
+
+  public void unregister(String jarName) {
+drillFuncRegistry.unregister(jarName);
+  }
+
+  /**
+   * Loads all missing functions from remote registry.
+   * Compares list of already registered jars and remote jars, loads 
missing jars.
+   * Missing jars are stores in local DRILL_UDF_DIR.
--- End diff --

Yes, I'll add note that it's an environmental variable but after chnages it 
will be $DRILL_CONF_DIR/udf.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473908#comment-15473908
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78008586
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
 ---
@@ -86,6 +108,7 @@ public FunctionImplementationRegistry(DrillConfig 
config, ScanResult classpathSc
   }
 }
 logger.info("Function registry loaded.  {} functions loaded in {} 
ms.", drillFuncRegistry.size(), w.elapsed(TimeUnit.MILLISECONDS));
+this.remoteFunctionRegistry = new RemoteFunctionRegistry(new 
UnregistrationListener());
--- End diff --

I figured out that not, since it's just class initialization and it doesn't 
load any functions.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473894#comment-15473894
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r78007166
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -64,62 +76,134 @@
   .put("CONVERT_FROM", Pair.of(2, 2))
   .put("FLATTEN", Pair.of(1, 1)).build();
 
+  /** Registers all functions present in Drill classpath on start-up. All 
functions will be marked as built-in.*/
   public DrillFunctionRegistry(ScanResult classpathScan) {
+validate(BUILT_IN, classpathScan);
+register(BUILT_IN, classpathScan, this.getClass().getClassLoader());
+if (logger.isTraceEnabled()) {
+  StringBuilder allFunctions = new StringBuilder();
+  for (DrillFuncHolder method: 
registryHolder.getAllFunctionsWithHolders().values()) {
+allFunctions.append(method.toString()).append("\n");
+  }
+  logger.trace("Registered functions: [\n{}]", allFunctions);
+}
+  }
+
+  /**
+   * Validates all functions, present in jars.
+   * Will throw {@link FunctionValidationException} if:
+   * 1. Jar with the same name has been already registered.
+   * 2. Conflicting function with the similar signature is found.
+   * 3. Aggregating function is not deterministic.
+   *
+   * @return list of validated functions
+   */
+  public List validate(String jarName, ScanResult classpathScan) {
+List functions = Lists.newArrayList();
 FunctionConverter converter = new FunctionConverter();
 List providerClasses = 
classpathScan.getAnnotatedClasses();
 
-// Hash map to prevent registering functions with exactly matching 
signatures
-// key: Function Name + Input's Major Type
-// value: Class name where function is implemented
-//
-final Map functionSignatureMap = new HashMap<>();
+if (registryHolder.containsJar(jarName)) {
+  throw new FunctionValidationException(String.format("Jar %s is 
already registered", jarName));
+}
+
+final ListMultimap allFuncWithSignatures = 
registryHolder.getAllFunctionsWithSignatures();
+
 for (AnnotatedClassDescriptor func : providerClasses) {
-  DrillFuncHolder holder = converter.getHolder(func);
+  DrillFuncHolder holder = converter.getHolder(func, 
ClassLoader.getSystemClassLoader());
   if (holder != null) {
-// register handle for each name the function can be referred to
-String[] names = holder.getRegisteredNames();
 
-// Create the string for input types
 String functionInput = "";
+List types = Lists.newArrayList();
 for (DrillFuncHolder.ValueReference ref : holder.parameters) {
   functionInput += ref.getType().toString();
+  types.add(ref.getType());
 }
+
+String[] names = holder.getRegisteredNames();
 for (String name : names) {
   String functionName = name.toLowerCase();
-  registeredFunctions.put(functionName, holder);
   String functionSignature = functionName + functionInput;
-  String existingImplementation;
-  if ((existingImplementation = 
functionSignatureMap.get(functionSignature)) != null) {
-throw new AssertionError(
-String.format(
-"Conflicting functions with similar signature found. 
Func Name: %s, Class name: %s " +
-" Class name: %s", functionName, func.getClassName(), 
existingImplementation));
-  } else if (holder.isAggregating() && !holder.isDeterministic() ) 
{
-logger.warn("Aggregate functions must be deterministic, did 
not register function {}", func.getClassName());
+
+  if 
(allFuncWithSignatures.get(functionName).contains(functionSignature)) {
+throw new FunctionValidationException(
+String.format("Conflicting function with similar signature 
found. " +
+"Function name: %s, class name: %s, input 
parameters : %s",
+functionName, func.getClassName(), functionInput));
+  } else if (holder.isAggregating() && !holder.isDeterministic()) {
+throw new FunctionValidationException(
+String.format("Aggregate functions must be deterministic: 
%s", func.getClassName()));
   } else {
-functionSignatureMap.put(functionSignature, 
func.getClassName());
+

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473435#comment-15473435
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77977654
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillSimpleFuncHolder.java
 ---
@@ -40,10 +40,12 @@
   static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(DrillSimpleFuncHolder.class);
 
   private final String drillFuncClass;
+  private final ClassLoader classLoader;
--- End diff --

Agree.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473431#comment-15473431
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77977432
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -218,4 +302,141 @@ private void 
registerOperatorsWithoutInference(DrillOperatorTable operatorTable)
   }
 }
   }
+
+  /**
+   * Function registry holder. Stores function implementations by jar 
name, function name.
+   */
+  private class GenericRegistryHolder {
+private final ReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
+private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
+
+// jar name, Map
+private final Map> jars;
+
+// function name, Map
+private final Map> functions;
+
+public GenericRegistryHolder() {
+  this.functions = Maps.newHashMap();
+  this.jars = Maps.newHashMap();
+}
+
+public void addJar(T jName, Map> sNameMap) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+Map map = jars.get(jName);
+if (map != null) {
+  removeAllByJar(jName);
+}
+map = Maps.newHashMap();
+jars.put(jName, map);
+
+for (Entry> entry : sNameMap.entrySet()) {
+  T sName = entry.getKey();
+  Pair pair = entry.getValue();
+  addFunction(jName, pair.getKey(), sName, pair.getValue());
+}
+  }
+}
+
+public void removeJar(T jName) {
+  try (AutoCloseableLock lock = writeLock.open()) {
+removeAllByJar(jName);
+  }
+}
+
+public List getAllJarNames() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return Lists.newArrayList(jars.keySet());
+  }
+}
+
+public List getAllFunctionNames(T jName) {
+  try  (AutoCloseableLock lock = readLock.open()){
+Map map = jars.get(jName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.keySet());
+  }
+}
+
+public ListMultimap getAllFunctionsWithHolders() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().values()));
+}
+return multimap;
+  }
+}
+
+public ListMultimap getAllFunctionsWithSignatures() {
+  try (AutoCloseableLock lock = readLock.open()) {
+ListMultimap multimap = ArrayListMultimap.create();
+for (Entry> entry : functions.entrySet()) {
+  multimap.putAll(entry.getKey(), 
Lists.newArrayList(entry.getValue().keySet()));
+}
+return multimap;
+  }
+}
+
+public List getHoldersByFunctionName(T fName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+Map map = functions.get(fName);
+return map == null ? Lists.newArrayList() : 
Lists.newArrayList(map.values());
+  }
+}
+
+public boolean containsJar(T jName) {
+  try (AutoCloseableLock lock = readLock.open()) {
+return jars.containsKey(jName);
+  }
+}
+
+public int functionsSize() {
+  try (AutoCloseableLock lock = readLock.open()) {
+return functions.size();
+  }
+}
+
+private void addFunction(T jName, T fName, T sName, U fHolder) {
--- End diff --

sName stands for signature name but I'll add comments.


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA

[jira] [Commented] (DRILL-4726) Dynamic UDFs support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473422#comment-15473422
 ] 

ASF GitHub Bot commented on DRILL-4726:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/574#discussion_r77976915
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
 ---
@@ -64,62 +76,134 @@
   .put("CONVERT_FROM", Pair.of(2, 2))
   .put("FLATTEN", Pair.of(1, 1)).build();
 
+  /** Registers all functions present in Drill classpath on start-up. All 
functions will be marked as built-in.*/
   public DrillFunctionRegistry(ScanResult classpathScan) {
+validate(BUILT_IN, classpathScan);
+register(BUILT_IN, classpathScan, this.getClass().getClassLoader());
+if (logger.isTraceEnabled()) {
+  StringBuilder allFunctions = new StringBuilder();
+  for (DrillFuncHolder method: 
registryHolder.getAllFunctionsWithHolders().values()) {
+allFunctions.append(method.toString()).append("\n");
+  }
+  logger.trace("Registered functions: [\n{}]", allFunctions);
+}
+  }
+
+  /**
+   * Validates all functions, present in jars.
+   * Will throw {@link FunctionValidationException} if:
--- End diff --

Agree


> Dynamic UDFs support
> 
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
>  Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)