Hi,
I could get some steps forward, I recompiled Zeppelin for Hadoop 2.6,
seems that S3 has only been fixed there, but now I have the next
Problem :
java.io.IOException: No FileSystem for scheme: s3n at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
I've found following workaround, but I'm not able to transpose the
steps for Zeppelin, can anybody help ? :
1. fs.s3n.impl
> We added a property to the core-site.xml file:
You don't need to explicitly set this. It has never been done so in
previous versions.
Take a look at FileSystem#loadFileSystem, which is called from
FileSystem#getFileSystemClass.
Subclasses of FileSystem are loaded automatically if they are
available on a classloader you care.
So you just need to make sure hadoop-aws.jar is on a classpath.
For file system shell, this is done in hadoop-env.sh,
while for a MR job, in mapreduce.application.classpath,
or for YARN, in yarn.application.classpath.
2. mapreduce.application.classpath
> And updated the classpath for mapreduce applications:
Note that it points to a distributed cache on the default HDP 2.2 distribution.
<property>
<name>mapreduce.application.classpath</name>
<value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
</property>
* $PWD/mr-framework/hadoop/share/hadoop/tools/lib/* contains
hadoop-aws.jar(S3NFileSystem)
While on a vanilla hadoop, it looks like standard paths as yours.
<property>
<name>mapreduce.application.classpath</name>
<value>/hadoop-2.6.0/etc/hadoop:/hadoop-2.6.0/share/hadoop/common/lib/*:/hadoop-2.6.0/share/hadoop/common/*:/hadoop-2.6.0/share/hadoop/hdfs:/hadoop-2.6.0/share/hadoop/hdfs/lib/*:/hadoop-2.6.0/share/hadoop/hdfs/*:/hadoop-2.6.0/share/hadoop/yarn/lib/*:/hadoop-2.6.0/share/hadoop/yarn/*:/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:/hadoop-2.6.0/share/hadoop/mapreduce/*:/hadoop-2.6.0/contrib/capacity-scheduler/*.jar:/hadoop-2.6.0/share/hadoop/tools/lib/*</value>
</property>
Thanks,
Sato
2015-07-16 11:03 GMT+02:00 Schmirr Wurst <schmirrwu...@gmail.com>:
> Hi,
>
> I've just try to access S3 from zeppelin, but it seems I'm missing
> some libraries ?
>
> Can somebody tell me what libraries I have to install were ? I'm not
> familiar with java...
>
> Best.
> SM
