[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438068#comment-16438068
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user ChinmaySKulkarni closed the pull request at:

https://github.com/apache/phoenix/pull/292


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Josh Elser (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410338#comment-16410338
 ] 

Josh Elser commented on PHOENIX-4231:
-

Just noticed this via some broken tests at $dayjob. Nice change to stumble upon 
though. Any chance you could update [https://phoenix.apache.org/tuning.html] 
with this new configuration property, [~ckulkarni]? :)

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399883#comment-16399883
 ] 

Hudson commented on PHOENIX-4231:
-

ABORTED: Integrated in Jenkins build Phoenix-4.x-HBase-1.3 #63 (See 
[https://builds.apache.org/job/Phoenix-4.x-HBase-1.3/63/])
PHOENIX-4231 Support restriction of remote UDF load sources (apurtell: rev 
ae3618ff88c36eb04734fad78ac64c8989fc470f)
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/UDFExpression.java
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/UserDefinedFunctionsIT.java


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399845#comment-16399845
 ] 

Hudson commented on PHOENIX-4231:
-

FAILURE: Integrated in Jenkins build PreCommit-PHOENIX-Build #1805 (See 
[https://builds.apache.org/job/PreCommit-PHOENIX-Build/1805/])
PHOENIX-4231 Support restriction of remote UDF load sources (apurtell: rev 
74228aee724e24ddb00bef2be0c7430172b699a8)
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/UserDefinedFunctionsIT.java
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/UDFExpression.java


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399635#comment-16399635
 ] 

Hudson commented on PHOENIX-4231:
-

SUCCESS: Integrated in Jenkins build Phoenix-4.x-HBase-0.98 #1835 (See 
[https://builds.apache.org/job/Phoenix-4.x-HBase-0.98/1835/])
PHOENIX-4231 Support restriction of remote UDF load sources (apurtell: rev 
ade93c9d5ac6ecad2234d22da6fdbb1168c5d32a)
* (edit) 
phoenix-core/src/it/java/org/apache/phoenix/end2end/UserDefinedFunctionsIT.java
* (edit) 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/UDFExpression.java


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 4.14.0
>
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399219#comment-16399219
 ] 

Ethan Wang commented on PHOENIX-4231:
-

[~apurtell]

Thanks!

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399114#comment-16399114
 ] 

Andrew Purtell commented on PHOENIX-4231:
-

Ok

Committing today.

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393524#comment-16393524
 ] 

Ethan Wang commented on PHOENIX-4231:
-

[~ckulkarni] thanks. done with Phoenix-4231-v2.patch.

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231-v2.patch, PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Chinmay Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393489#comment-16393489
 ] 

Chinmay Kulkarni commented on PHOENIX-4231:
---

[~aertoria] please remove unnecessary changes to files MutationState.java and 
PhoenixStatement.java. Everything else lgtm.

 

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393477#comment-16393477
 ] 

Ethan Wang commented on PHOENIX-4231:
-

Thanks [~apurtell]

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393458#comment-16393458
 ] 

Andrew Purtell commented on PHOENIX-4231:
-

+1

Any concerns about commit? If not I will do so shortly.

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379466#comment-16379466
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on the issue:

https://github.com/apache/phoenix/pull/292
  
Sent out another patch for review. Let us close this p.r.


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379456#comment-16379456
 ] 

Ethan Wang commented on PHOENIX-4231:
-

Please review the patch. [~apurtell] [~rajeshbabu]
{quote}Either way, we want UDF loading to be restricted to one place only.
{quote}
This patch basically does exactly this.

FYI [~ckulkarni]

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
> Attachments: PHOENIX-4231.patch
>
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377696#comment-16377696
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r170755367
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
--- End diff --

The comment "Note that this does not support HDFS URIs without scheme and 
authority." I wonder where did you discover that.

If I'm not mistaken FileUtil.copy internally calls fs.copyFromLocalFile 
without doing file.islocal check.


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377520#comment-16377520
 ] 

Ethan Wang commented on PHOENIX-4231:
-

To summarize what we are going to be working on this jira: As of today, loading 
from add jars 'XX' allows loading the jars from any hdfs network location to 
local's hbase.dynamic.jars.dir. So we are now going to make addition features 
to PHOENIX-1890 so that:

when a property, say "loading.from.network.allowed" property is configured as 
false, "add jars" command only restricted to load from local file system or 
local hdfs.

Is the plan above accurate?

[~rajeshbabu]

[~apurtell]

Once this feature is done. user will be doing the following for creating a Udf:

Add jars hdfs://localhost/udf.jar

CREATE FUNCTION func(VARCHAR) returns VARCHAR as 'com.example.test.udfFunc' 
using jar 'configured/hbase/dynamic/jars/dir/udf2.jar'

 

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375065#comment-16375065
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r170387933
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/expression/function/UDFExpression.java
 ---
@@ -198,18 +199,26 @@ public static DynamicClassLoader getClassLoader(final 
PName tenantId, final Stri
 }
 return cl;
 } else {
-cl = pathSpecificCls.get(jarPath);
-if (cl == null) {
-Configuration conf = HBaseConfiguration.create(config);
-conf.set(DYNAMIC_JARS_DIR_KEY, parent);
-cl = new DynamicClassLoader(conf, 
UDFExpression.class.getClassLoader());
-}
-// Cache class loader as a weak value, will be GC'ed when no 
reference left
-DynamicClassLoader prev = pathSpecificCls.putIfAbsent(jarPath, 
cl);
-if (prev != null) {
-cl = prev;
+String rawPath = new 
Path(config.get(DYNAMIC_JARS_DIR_KEY)).toUri().getRawPath();
+// jarPath is provided as an HDFS URI without scheme and 
authority, but the jar is inside the configured hbase.dynamic.jars.dir
+if(rawPath.equals(parent)) {
--- End diff --

This if block seems will never be reachable. See line 188.  Aren't these 
two condition equivalent?


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370810#comment-16370810
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r169509766
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
--- End diff --

This copy happen at phoenixStatment.java which is at client side. Shouldn't 
the checking be happening at Server side


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369703#comment-16369703
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r169213703
  
--- Diff: 
phoenix-core/src/it/java/org/apache/phoenix/end2end/UserDefinedFunctionsIT.java 
---
@@ -322,6 +327,99 @@ public void testDeleteJar() throws Exception {
 assertFalse(rs.next());
 }
 
+/**
+ * Test adding jars from an HDFS URI.
+ * @throws Exception
+ */
+@Test
+public void testAddJarsFromHDFS() throws Exception {
+compileTestClass(MY_ARRAY_INDEX_CLASS_NAME, 
MY_ARRAY_INDEX_PROGRAM, 7);
+Statement stmt = driver.connect(url, 
EMPTY_PROPS).createStatement();
+// Note that we have already added all locally created UDF jars to 
the hbase.dynamic.jars.dir directory
+ResultSet rs = stmt.executeQuery("list jars");
+int count = 0;
+while(rs.next()) {
+count++;
+}
+Path destJarPathOnHDFS = 
copyJarsFromDynamicJarsDirToDummyHDFSDir("myjar7.jar");
+stmt.execute("delete jar '"+ 
util.getConfiguration().get(QueryServices.DYNAMIC_JARS_DIR_KEY)+"/"+"myjar7.jar'");
+stmt.execute("add jars '" + destJarPathOnHDFS.toString() + "'");
+rs = stmt.executeQuery("list jars");
+int finalCount = 0;
+while(rs.next()) {
+finalCount++;
+}
+assertEquals(count, finalCount);
--- End diff --

original `list jars` equal to `list jars`, shouldn't the newly added jar 
make it plus 1? i.e.,
 assertEquals(count+1, finalCount);


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Rajeshbabu Chintaguntla (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361869#comment-16361869
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-4231:
--

Thanks [~apurtell] for details.
bq. we could define a Phoenix specific directory for UDFs, configurable by site 
file.
This is better  and we can restrict loading to one place only.

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361660#comment-16361660
 ] 

Andrew Purtell commented on PHOENIX-4231:
-

No, we want to restrict loading to hbase.dynamic.jars.dir. Or we could define a 
Phoenix specific directory for UDFs, configurable by site file.

Whether or not the user can write to the directory isn't important. In fact, in 
our production we don't want the user to be able to write jars to HDFS at all. 
We will preposition safe and vetted UDF jars in HDFS and Phoenix can load them 
from there, but ONLY from there.

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Rajeshbabu Chintaguntla (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360531#comment-16360531
 ] 

Rajeshbabu Chintaguntla commented on PHOENIX-4231:
--

[~aertoria] [~ckulkarni]
As for my understanding in the pull request we are restricting the udf jars 
path to hbase.dynamic.jars.dir path. But some cases as [~apurtell] mentioned 
all clients may not allowed for reading or writing jars to 
hbase.dynamic.jars.dir path in that case we can make use any other directory in 
HDFS for keeping the jars. So I think we should not impose this restriction 
otherwise UDFs feature may not be usable until unless the client have write 
permissions on hbase.dynamic.jars.dir path.
[~apurtell] 
As for the description what I think is currently we are able to load jars from 
any HDFS so we need to restrict udf jars to to be loaded from local HDFS only 
right?


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Ethan Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354825#comment-16354825
 ] 

Ethan Wang commented on PHOENIX-4231:
-

{quote}The setting hbase.dynamic.jars.dir can be used to restrict locations for 
jar loading but is only applied to jars loaded from the local filesystem.
{quote}
So as of today, what configuration that user need to set for 
hbase.dynamic.jars.dir, in order to restrict that only the jar from the local 
filesystem (not the network) is able to load in ?

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354806#comment-16354806
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r166491289
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
--- End diff --

@apurtell 
Since @ChinmaySKulkarni will be absent for a while, I will be wrap up this 
patch for the team.
do you know how is hbase.dynamic.jars.dir used differently than 
hbase.local.jars.dir


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354789#comment-16354789
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user aertoria commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r166487244
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
+false, true, conf);
--- End diff --

@ChinmaySKulkarni 
How is Hadoop.fileUtil.copy different from filesystem.copy


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354739#comment-16354739
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user apurtell commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r166479944
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
--- End diff --

Actually, doesn't this imply the client should have write perms to 
hbase.dynamic.jars.dir? We don't want to allow arbitrary clients to write 
there. 


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354737#comment-16354737
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user apurtell commented on a diff in the pull request:

https://github.com/apache/phoenix/pull/292#discussion_r166479267
  
--- Diff: 
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixStatement.java ---
@@ -907,10 +909,15 @@ public MutationState execute() throws SQLException {
 try {
 FileSystem fs = 
dynamicJarsDirPath.getFileSystem(conf);
 List jarPaths = getJarPaths();
-for (LiteralParseNode jarPath : jarPaths) {
-File f = new File((String) jarPath.getValue());
-fs.copyFromLocalFile(new 
Path(f.getAbsolutePath()), new Path(
-dynamicJarsDir + f.getName()));
+for (LiteralParseNode jarPathNode : jarPaths) {
+  String jarPathName = (String) 
jarPathNode.getValue();
+  File f = new File(jarPathName);
+  Path dynamicJarsDirPathWithJar = new 
Path(dynamicJarsDir + f.getName());
+  // Copy the jar (can be local or on HDFS) to the 
hbase.dynamic.jars.dir directory.
+  // Note that this does not support HDFS URIs 
without scheme and authority.
+  Path jarPath = new Path(jarPathName);
+  FileUtil.copy(jarPath.getFileSystem(conf), 
jarPath, fs, dynamicJarsDirPathWithJar,
--- End diff --

If the client does not have perms to write to hbase.dynamic.jars.dir (and I 
expect normally clients will not have write perms to this directory, only admin 
clients will have it), the copy will fail and throw an IOException. The result 
may not be user friendly, though. Did you try this? What happens? 


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350953#comment-16350953
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user JamesRTaylor commented on the issue:

https://github.com/apache/phoenix/pull/292
  
Thanks for the patch, @ChinmaySKulkarni. Best person to review is 
@chrajeshbabu who originally added support for UDFs.


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350911#comment-16350911
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

Github user ChinmaySKulkarni commented on the issue:

https://github.com/apache/phoenix/pull/292
  
@apurtell @twdsilva @jtaylor-sfdc please review. Thanks.


> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350909#comment-16350909
 ] 

ASF GitHub Bot commented on PHOENIX-4231:
-

GitHub user ChinmaySKulkarni opened a pull request:

https://github.com/apache/phoenix/pull/292

PHOENIX-4231: Support restriction of remote UDF load sources

- Added feature to be able to add jars from an HDFS URI.
- Restrict loading of jars to be only from the hbase.dynamic.jars.dir
directory.

Testing done:
- Tested that the user is able to add jars from an HDFS URI reachable on the
network as well as local filesystem.
- Tested that the user is unable to create a function where the jar is
loaded from any directory apart from hbase.dynamic.jars.dir.
- Tested that HDFS URIs without scheme and authority work.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ChinmaySKulkarni/phoenix PHOENIX-4231

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/phoenix/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #292


commit 9351e1b4d4a5741bbf063f0d6cd31f14cfa1e6b6
Author: Chinmay Kulkarni 
Date:   2018-02-02T20:16:47Z

PHOENIX-4231: Support restriction of remote UDF load sources

- Added feature to be able to add jars from an HDFS URI.
- Restrict loading of jars to be only from the hbase.dynamic.jars.dir
directory.

Testing done:
- Tested that the user is able to add jars from an HDFS URI reachable on the
network as well as local filesystem.
- Tested that the user is unable to create a function where the jar is
loaded from any directory apart from hbase.dynamic.jars.dir.
- Tested that HDFS URIs without scheme and authority work.




> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (PHOENIX-4231) Support restriction of remote UDF load sources

[Chinmay Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/PHOENIX-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349314#comment-16349314
 ] 

Chinmay Kulkarni commented on PHOENIX-4231:
---

When users execute a _CREATE FUNCTION_ statement, here are the following 
options based on the UDF's jar path:
 # If the user does not provide the location of the jar file with 'using jar', 
the DynamicClassLoader will automatically try to load the class from the 
hbase.dynamic.jars.dir directory.
 # As of now, if the user provides an HDFS URI path for the jar file with 
'using jar ', the DynamicClassLoader will try to load the 
class from this location, without restricting the search to within 
hbase.dynamic.jars.dir. 
 # If the user wishes to load a local jar, he/she is expected to manually load 
the jar onto any HDFS filesystem reachable on the network (any such location, 
not necessarily restricted to hbase.dynamic.jars.dir) and follow step 2.
 # If the user calls 'add jar ' prior to the _CREATE FUNCTION_ 
statement, his/her local jar will be automatically copied to the 
hbase.dynamic.jars.dir directory. Note that this only applies to a local jar, 
not any jar on the HDFS. In this case, the user can follow step 1 or 2.

To support this feature, we can do the following:
 # Allow the user to 'add jar ' so their jar to be loaded 
is inside the hbase.dynamic.jars.dir directory. Currently, we only allow local 
jars to be added.
 # Load the class only from the hbase.dynamic.jars.dir directory, handling URIs 
without scheme and authority carefully.

Any suggestions or comments [~apurtell] [~jamestaylor]? 

 

> Support restriction of remote UDF load sources 
> ---
>
> Key: PHOENIX-4231
> URL: https://issues.apache.org/jira/browse/PHOENIX-4231
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Chinmay Kulkarni
>Priority: Major
>
> When allowUserDefinedFunctions is true, users can load UDFs remotely via a 
> jar file from any HDFS filesystem reachable on the network. The setting 
> hbase.dynamic.jars.dir can be used to restrict locations for jar loading but 
> is only applied to jars loaded from the local filesystem.  We should 
> implement support for similar restriction via configuration for jars loaded 
> via hdfs:// URIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)