Bernd, To answer your question about the HDFS version, I went back and looked at the patches I supplied for VFS-530. In the patches that I supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only changes were in the pom. This should mean that, for the parts of the Hadoop client objects that I am using, they are API compatible.
For the example, the list of jars depends on the Hadoop version. I think one way to handle that is to use the `hadoop classpath` command to create the classpath. The example would then look like: REP=~/.m2/repository HADOOP_HOME=<PATH TO HADOOP> HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar LIBS=$LIBS:$HADOOP_CLASSPATH java -cp $LIBS org.apache.commons.vfs2.example.Shell On my local machine, this looks like: 08:03:17 ~/EclipseWorkspace/commons-vfs2-project$ unset REP 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_HOME 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset LIBS 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ REP=~/.m2/repository 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_HOME=/home/dave/Software/hadoop-2.6.0 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:$HADOOP_CLASSPATH 08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ java -cp $LIBS org.apache.commons.vfs2.example.Shell 15/01/09 20:03:18 INFO impl.StandardFileSystemManager: Using "/tmp/vfs_cache" as temporary files store. VFS Shell 2.1-SNAPSHOT > info Default manager: "org.apache.commons.vfs2.impl.StandardFileSystemManager" version 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps, ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear, jar, sar, ejb3, tar, tbz2, tgz] Is this sufficient? Dave ----- Original Message ----- From: "Bernd Eckenfels" <e...@zusammenkunft.net> To: dlmar...@comcast.net Cc: "Commons Developers List" <dev@commons.apache.org> Sent: Friday, January 9, 2015 9:05:55 AM Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0 Hello Dave, for the download page (staged: http://people.apache.org/~ecki/commons-vfs/download.html) I need a list of libraries needed to run VFS to access hdfs. Could you maybe produce an example VFS Shell session similiar to https://wiki.apache.org/commons/VfsReleaseState where you list the command line needed. It would be best if this is an public HDFS instance (but I guess there is no such thing?) BTW: I had asked in the VFS-530 bug about how commonplace the different hdfs APIs are, and if it is really good if we bump the minimum version. As I understand it. Will HDFS with 1.1.2 be able to communicate with 2.6 instances? If yes I would prefer to keep it at 1.2 in this release. WDYT? Gruss Bernd
