[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743664#comment-15743664
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/669


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736769#comment-15736769
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/669
  
+1


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721909#comment-15721909
 ] 

Arina Ielchiieva commented on DRILL-5044:
-

Adding label ready-for-commit, since PR was approved by Paul in PR -> Thanks 
for making the changes. Looks good.
Unfortunately, Paul comment as not reflected in Jira comments automatically, to 
see that PR was actually approved, please refer to PR comments.

> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>  Labels: ready-to-commit
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712408#comment-15712408
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/669
  
@paul-rogers, I have made changes after CR, please review when possible.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705310#comment-15705310
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89994171
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -175,22 +175,20 @@ private void initRemoteRegistration(List 
functions,
 List remoteJars = 
remoteRegistry.getRegistry(version).getJarList();
 validateAgainstRemoteRegistry(remoteJars, jarManager.getBinaryName(), 
functions);
 jarManager.copyToRegistryArea();
-boolean cleanUp = true;
 List jars = Lists.newArrayList(remoteJars);
 
jars.add(Jar.newBuilder().setName(jarManager.getBinaryName()).addAllFunctionSignature(functions).build());
 Registry updatedRegistry = 
Registry.newBuilder().addAllJar(jars).build();
 try {
   remoteRegistry.updateRegistry(updatedRegistry, version);
-  cleanUp = false;
 } catch (VersionMismatchException ex) {
+  jarManager.deleteQuietlyFromRegistryArea();
--- End diff --

1. I guess having fixed number of retries is enough. Having retry and wait 
logic, may lead us to the point where user will have to wait for a long time 
till registration completes in case of busy system. With only retry logic we 
notify user pretty quickly that the system is busy and it's up to the user to 
decide when to try to register the function again,

2. Totally agree about recursion, since user may modify number of retry 
attempts, it's much better to have while loop to avoid stack overflow.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705314#comment-15705314
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89993367
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -175,22 +175,20 @@ private void initRemoteRegistration(List 
functions,
 List remoteJars = 
remoteRegistry.getRegistry(version).getJarList();
 validateAgainstRemoteRegistry(remoteJars, jarManager.getBinaryName(), 
functions);
 jarManager.copyToRegistryArea();
-boolean cleanUp = true;
 List jars = Lists.newArrayList(remoteJars);
 
jars.add(Jar.newBuilder().setName(jarManager.getBinaryName()).addAllFunctionSignature(functions).build());
 Registry updatedRegistry = 
Registry.newBuilder().addAllJar(jars).build();
 try {
   remoteRegistry.updateRegistry(updatedRegistry, version);
-  cleanUp = false;
 } catch (VersionMismatchException ex) {
+  jarManager.deleteQuietlyFromRegistryArea();
   if (retryAttempts-- == 0) {
 throw new DrillRuntimeException("Failed to update remote function 
registry. Exceeded retry attempts limit.");
   }
   initRemoteRegistration(functions, jarManager, remoteRegistry, 
retryAttempts);
-} finally {
-  if (cleanUp) {
-jarManager.deleteQuietlyFromRegistryArea();
-  }
+} catch (Exception e) {
--- End diff --

You are right. Updated the code, so now we delete jars only once on error.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705312#comment-15705312
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89993422
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -143,7 +143,7 @@ private Jar unregister(String jarName, 
RemoteFunctionRegistry remoteFunctionRegi
 if (retryAttempts-- == 0) {
   throw new DrillRuntimeException("Failed to update remote 
function registry. Exceeded retry attempts limit.");
 }
-unregister(jarName, remoteFunctionRegistry, retryAttempts);
+return unregister(jarName, remoteFunctionRegistry, retryAttempts);
--- End diff --

Agree. Done.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705311#comment-15705311
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89995323
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -271,6 +271,75 @@ public void testDuplicatedFunctionsInLocalRegistry() 
throws Exception {
   }
 
   @Test
+  public void testSuccessfulRegistrationAfterSeveralRetryAttempts() throws 
Exception {
+RemoteFunctionRegistry remoteFunctionRegistry = 
spyRemoteFunctionRegistry();
+copyDefaultJarsToStagingArea();
+
+doThrow(new VersionMismatchException("Version mismatch detected", 1))
+.doThrow(new VersionMismatchException("Version mismatch 
detected", 1))
+.doCallRealMethod()
+
.when(remoteFunctionRegistry).updateRegistry(any(Registry.class), 
any(DataChangeVersion.class));
+
+String summary = "The following UDFs in jar %s have been 
registered:\n" +
+"[custom_lower(VARCHAR-REQUIRED)]";
+
+testBuilder()
+.sqlQuery("create function using jar '%s'", 
default_binary_name)
+.unOrdered()
+.baselineColumns("ok", "summary")
+.baselineValues(true, String.format(summary, 
default_binary_name))
+.go();
+
+verify(remoteFunctionRegistry, times(3))
+.updateRegistry(any(Registry.class), 
any(DataChangeVersion.class));
+
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+assertFalse("Staging area should be empty", 
fs.listFiles(remoteFunctionRegistry.getStagingArea(), false).hasNext());
+assertFalse("Temporary area should be empty", 
fs.listFiles(remoteFunctionRegistry.getTmpArea(), false).hasNext());
+
+assertTrue("Binary should be present in registry area",
+fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), 
default_binary_name)));
+assertTrue("Source should be present in registry area",
+fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), 
default_source_name)));
+
+Registry registry = remoteFunctionRegistry.getRegistry();
+assertEquals("Registry should contain one jar", 
registry.getJarList().size(), 1);
+assertEquals(registry.getJar(0).getName(), default_binary_name);
+  }
+
+  @Test
+  public void testSuccessfulUnregistrationAfterSeveralRetryAttempts() 
throws Exception {
+RemoteFunctionRegistry remoteFunctionRegistry = 
spyRemoteFunctionRegistry();
+copyDefaultJarsToStagingArea();
+test("create function using jar '%s'", default_binary_name);
+
+reset(remoteFunctionRegistry);
+doThrow(new VersionMismatchException("Version mismatch detected", 1))
--- End diff --

It's Mockito functionality. You can mock the method to return failure or 
any result when it is being called.
In this case we mock `updateRegistry()` method to return 
VersionMismatchException first two times. This way we simulate the situation 
that someone has updated remote function registry before us.
After that we instruct to call real method.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702753#comment-15702753
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89848425
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DropFunctionHandler.java
 ---
@@ -143,7 +143,7 @@ private Jar unregister(String jarName, 
RemoteFunctionRegistry remoteFunctionRegi
 if (retryAttempts-- == 0) {
   throw new DrillRuntimeException("Failed to update remote 
function registry. Exceeded retry attempts limit.");
 }
-unregister(jarName, remoteFunctionRegistry, retryAttempts);
+return unregister(jarName, remoteFunctionRegistry, retryAttempts);
--- End diff --

Comments from registration process apply here as well.


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702750#comment-15702750
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89848865
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/TestDynamicUDFSupport.java ---
@@ -271,6 +271,75 @@ public void testDuplicatedFunctionsInLocalRegistry() 
throws Exception {
   }
 
   @Test
+  public void testSuccessfulRegistrationAfterSeveralRetryAttempts() throws 
Exception {
+RemoteFunctionRegistry remoteFunctionRegistry = 
spyRemoteFunctionRegistry();
+copyDefaultJarsToStagingArea();
+
+doThrow(new VersionMismatchException("Version mismatch detected", 1))
+.doThrow(new VersionMismatchException("Version mismatch 
detected", 1))
+.doCallRealMethod()
+
.when(remoteFunctionRegistry).updateRegistry(any(Registry.class), 
any(DataChangeVersion.class));
+
+String summary = "The following UDFs in jar %s have been 
registered:\n" +
+"[custom_lower(VARCHAR-REQUIRED)]";
+
+testBuilder()
+.sqlQuery("create function using jar '%s'", 
default_binary_name)
+.unOrdered()
+.baselineColumns("ok", "summary")
+.baselineValues(true, String.format(summary, 
default_binary_name))
+.go();
+
+verify(remoteFunctionRegistry, times(3))
+.updateRegistry(any(Registry.class), 
any(DataChangeVersion.class));
+
+FileSystem fs = remoteFunctionRegistry.getFs();
+
+assertFalse("Staging area should be empty", 
fs.listFiles(remoteFunctionRegistry.getStagingArea(), false).hasNext());
+assertFalse("Temporary area should be empty", 
fs.listFiles(remoteFunctionRegistry.getTmpArea(), false).hasNext());
+
+assertTrue("Binary should be present in registry area",
+fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), 
default_binary_name)));
+assertTrue("Source should be present in registry area",
+fs.exists(new Path(remoteFunctionRegistry.getRegistryArea(), 
default_source_name)));
+
+Registry registry = remoteFunctionRegistry.getRegistry();
+assertEquals("Registry should contain one jar", 
registry.getJarList().size(), 1);
+assertEquals(registry.getJar(0).getName(), default_binary_name);
+  }
+
+  @Test
+  public void testSuccessfulUnregistrationAfterSeveralRetryAttempts() 
throws Exception {
+RemoteFunctionRegistry remoteFunctionRegistry = 
spyRemoteFunctionRegistry();
+copyDefaultJarsToStagingArea();
+test("create function using jar '%s'", default_binary_name);
+
+reset(remoteFunctionRegistry);
+doThrow(new VersionMismatchException("Version mismatch detected", 1))
--- End diff --

Perhaps a few words of explanation about what doThrow does?


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702754#comment-15702754
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/669#discussion_r89847859
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateFunctionHandler.java
 ---
@@ -175,22 +175,20 @@ private void initRemoteRegistration(List 
functions,
 List remoteJars = 
remoteRegistry.getRegistry(version).getJarList();
 validateAgainstRemoteRegistry(remoteJars, jarManager.getBinaryName(), 
functions);
 jarManager.copyToRegistryArea();
-boolean cleanUp = true;
 List jars = Lists.newArrayList(remoteJars);
 
jars.add(Jar.newBuilder().setName(jarManager.getBinaryName()).addAllFunctionSignature(functions).build());
 Registry updatedRegistry = 
Registry.newBuilder().addAllJar(jars).build();
 try {
   remoteRegistry.updateRegistry(updatedRegistry, version);
-  cleanUp = false;
 } catch (VersionMismatchException ex) {
+  jarManager.deleteQuietlyFromRegistryArea();
--- End diff --

I wonder about the retry approach. If we get a version mis-match, it means 
that something has changed: some other agent is modifying the registry at the 
same time. How should this be handed? Fixed number of retry? Wait and retry? 
Presumably, after some time, the system will get less busy, or we'll finally 
get our crack at the registry.

Also, is it wise to use recursion to retry? Doing so causes a stack depth 
proportional to the retry count, which may not be ideal. Loop instead?


> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Paul Rogers
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701703#comment-15701703
 ] 

ASF GitHub Bot commented on DRILL-5044:
---

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/669

DRILL-5044: After the dynamic registration of multiple jars simultane…

…ously not all UDFs were registered

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-5044

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/669.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #669


commit a5ee5549e834bfbf9142bfbf51ff37c421cc23db
Author: Arina Ielchiieva 
Date:   2016-11-25T16:44:08Z

DRILL-5044: After the dynamic registration of multiple jars simultaneously 
not all UDFs were registered




> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-5044) After the dynamic registration of multiple jars simultaneously not all UDFs were registered

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701673#comment-15701673
 ] 

Arina Ielchiieva commented on DRILL-5044:
-

The reason of this failure is that when update to remote UDFs registry failed 
with VersionMismatchException, retry logic was applied. Even though after 
retry, update was successful, jars were removed from remote UDFs area in 
finally block.

> After the dynamic registration of multiple jars simultaneously not all UDFs 
> were registered
> ---
>
> Key: DRILL-5044
> URL: https://issues.apache.org/jira/browse/DRILL-5044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Affects Versions: 1.9.0
>Reporter: Roman
>Assignee: Arina Ielchiieva
>
> I tried to register 21 jars simultaneously (property 'udf.retry-attempts' = 
> 30) and not all jars were registered. As I see in output, all function were 
> registered and /staging directory was empty, but not all of jars were moved 
> into /registry directory. 
> For example, after simultaneously registration I saw "The following UDFs in 
> jar test-1.1.jar have been registered: [test1(VARCHAR-REQUIRED)" message, but 
> this jar was not in /registry directory. When I tried to run function test1, 
> I got this error: "Error: SYSTEM ERROR: SqlValidatorException: No match found 
> for function signature test1()". And when I tried to reregister 
> this jar, I got "Jar with test-1.1.jar name has been already registered".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)