[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-14 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291840#comment-16291840
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/15/17 12:31 AM:
--

[~aw] Thanks for your comments.
bq. In hadoop 3.x when HADOOP_OPTIONAL_TOOLS are loaded, they expect their 
dependencies from the other directories to already be in the classpath. I'd be 
greatly surprised if they find them from the shaded jar since I'm pretty sure 
we do class hiding.

I have included Hadoop tools lib folder, so which will be adequate to get all 
its dependencies. I have just ran distcp for sample, need to run other commands 
to know more if they cause issue.

bq. But no matter what you do, everyone is already trained to use 'hadoop 
classpath'. Until that gets changed, anything else that is done is moot.

if I override all home variables(like hdfs_home, mapred_home) to same client 
location in hadoop-layout.sh, it will have only shaded client jars. so i think 
this way we can address hadoop classpath problem.





was (Author: bharatviswa):
bq. In hadoop 3.x when HADOOP_OPTIONAL_TOOLS are loaded, they expect their 
dependencies from the other directories to already be in the classpath. I'd be 
greatly surprised if they find them from the shaded jar since I'm pretty sure 
we do class hiding.

I have included Hadoop tools lib folder, so which will be adequate to get all 
its dependencies. I have just ran distcp for sample, need to run other commands 
to know more if they cause issue.

bq. But no matter what you do, everyone is already trained to use 'hadoop 
classpath'. Until that gets changed, anything else that is done is moot.

if I override all home variables(like hdfs_home, mapred_home) to same client 
location in hadoop-layout.sh, it will have only shaded client jars. so i think 
this way we can address hadoop classpath problem.




> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-14 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291717#comment-16291717
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/14/17 11:20 PM:
--

[~aw]

Ya admin commands will break with this, but hadoop tools commands should work 
as I have copied tools/lib folder and
changed hadoop-layout.sh accordingly


{code:java}
HADOOP_TOOLS_HOME=${HADOOP_HOME}
HADOOP_TOOLS_DIR=${HADOOP_TOOLS_DIR:-"tools"}
HADOOP_TOOLS_LIB_JARS_DIR=${HADOOP_TOOLS_LIB_JARS_DIR:-"${HADOOP_TOOLS_DIR}/lib"}
{code}

For backward compatibility we can have existing hdfs script as it is, and we 
can have 2 scripts 
1. One for client commands 
2. Other for admin/server end commands. 
So, that when some one does this kind of seperating shaded client jars, they 
can use respective client script. With this users will not get class not found 
exceptions for some classes and other strange errors. Thinking loud, not sure 
might be there is better approach than this.








was (Author: bharatviswa):
[~aw]

Ya admin commands will break with this, but hadoop tools commands should work 
as I have copied tools/lib folder and
changed hadoop-layout.sh accordingly


{code:java}
HADOOP_TOOLS_HOME=${HADOOP_HOME}
HADOOP_TOOLS_DIR=${HADOOP_TOOLS_DIR:-"tools"}
HADOOP_TOOLS_LIB_JARS_DIR=${HADOOP_TOOLS_LIB_JARS_DIR:-"${HADOOP_TOOLS_DIR}/lib"}
{code}

For backward compatibility we can have existing hdfs script as it is, and we 
can have 2 scripts 
1. One for client commands 
2. Other for admin commands. 
So, that when some one does this kind of seperating shaded client jars, they 
can use respective client script. With this users will not get class not found 
exceptions for some classes and other strange errors. Thinking loud, not sure 
might be there is better approach than this.







> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-13 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290391#comment-16290391
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/14/17 6:17 AM:
-

[~aw]
Thanks for your reply.

bq. a) What is the goal here?
The goal is to run hdfs commands with shaded client jars.

bq. b) What exact surgery was performed to make that happen, since that's not 
how Hadoop ships out of the box.
Changed the hadoop-layout.sh mapred_home hadoop_hdfs_home etc. to point to only 
hdfs client jar location and copied all tools and shaded client jars .

So, for this to work, we need to copy below jars in to clients location to make 
hdfs commands work then. Is this the solution.

Actually this is done as part of work to use shaded client jars for running 
hdfs commands and also not to expose other run time jars to clients. And also 
use same jars for running hdfs commands.


was (Author: bharatviswa):
[~aw]

bq. a) What is the goal here?
The goal is to run hdfs commands with shaded client jars.

bq. b) What exact surgery was performed to make that happen, since that's not 
how Hadoop ships out of the box.
Changed the hadoop-layout.sh mapred_home hadoop_hdfs_home etc. to point to only 
hdfs client jar location and copied all tools and shaded client jars .

So, for this to work, we need to copy below jars in to clients location to make 
hdfs commands work then. Is this the solution.

Actually this is done as part of work to use shaded client jars for running 
hdfs commands and also not to expose other run time jars to clients. And also 
use same jars for running hdfs commands.

> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-11 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286871#comment-16286871
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/12/17 12:42 AM:
--

After adding below jars to classpath, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar



was (Author: bharatviswa):
After adding below jars, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar


> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-11 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286871#comment-16286871
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/12/17 12:42 AM:
--

After adding below jars to classpath, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar


cc [~busbey]


was (Author: bharatviswa):
After adding below jars to classpath, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar


> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12916) HDFS commands throws error, when only shaded clients in classpath

2017-12-11 Thread Bharat Viswanadham (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286871#comment-16286871
 ] 

Bharat Viswanadham edited comment on HDFS-12916 at 12/12/17 12:54 AM:
--

After adding below jars to classpath, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar

so, do we need to add  above jars and shade them in  hadoop-client-runtime jar

cc [~busbey]


was (Author: bharatviswa):
After adding below jars to classpath, commands started working

commons-logging-1.1.3.jar
htrace-core4-4.1.0-incubating.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar


cc [~busbey]

> HDFS commands throws error, when only shaded clients in classpath
> -
>
> Key: HDFS-12916
> URL: https://issues.apache.org/jira/browse/HDFS-12916
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> [root@n001 hadoop]# bin/hdfs dfs -rm /
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/htrace/core/Tracer$Builder
>   at org.apache.hadoop.fs.FsShell.run(FsShell.java:303)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.htrace.core.Tracer$Builder
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 4 more
> cc [~busbey]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org