FWIW, I've not had luck getting VFS with Hadoop 1.1.2 to work on Windows. I run the unit tests on a Linux VM. YMMV.
Gary On Fri, Apr 11, 2014 at 5:07 PM, Roger Whitcomb <[email protected]>wrote: > My client is on Windows 7, but the servers (Hadoop clusters) are mostly > Ubuntu (Cloudera VMs) for now. > > There doesn't seem to be any problem of Hadoop versions (i.e., it finds > and loads the Hadoop classes just fine), but it isn't being told what the > right configuration is. > > I've now seem some other postings elsewhere that seem to indicate that the > FS.defaultFS setting in the Configuration object has to point to the remote > system. So, I'm guessing that the VFS HdfsProvider needs to know some more > than I am telling it. Or maybe this is a change from Hadoop 1.x to 2.x?! > > Thanks, > ~Roger > > -----Original Message----- > From: Gary Gregory [mailto:[email protected]] > Sent: Friday, April 11, 2014 12:58 PM > To: Commons Users List > Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > access? > > Curious: What OS are you on? > > We build trunk with Hadoop version 1.1.2, so who know what happens with a > newer version. > > I would start with trying to build VFS with the version of Hadoop you want > to use to see what happens... > > Gary > > > On Fri, Apr 11, 2014 at 3:51 PM, Roger Whitcomb > <[email protected]>wrote: > > > Hi Martin, > > Well, I downloaded Hadoop 2.3.0 from one of the mirrors, and cannot > > find a "hadoop-core....jar" file anywhere in that distribution. But I > > was able to track down sort of the minimum set of .jars necessary to > > at least (try to) connect using Commons VFS 2.1: > > commons-collections-3.2.1.jar > > commons-configuration-1.6.jar > > commons-lang-2.6.jar > > commons-vfs2-2.1-SNAPSHOT.jar > > guava-11.0.2.jar > > hadoop-auth-2.3.0.jar > > hadoop-common-2.3.0.jar > > log4j-1.2.17.jar > > slf4j-api-1.7.5.jar > > slf4j-log4j12-1.7.5.jar > > > > What's happening now is that I instantiated the HdfsProvider this way: > > private static DefaultFileSystemManager manager = null; > > > > static > > { > > manager = new DefaultFileSystemManager(); > > try { > > manager.setFilesCache(new DefaultFilesCache()); > > manager.addProvider("hdfs", new HdfsFileProvider()); > > manager.setFileContentInfoFactory(new > > FileContentInfoFilenameFactory()); > > manager.setFilesCache(new SoftRefFilesCache()); > > manager.setReplicator(new DefaultFileReplicator()); > > manager.setCacheStrategy(CacheStrategy.ON_RESOLVE); > > manager.init(); > > } > > catch (final FileSystemException e) { > > throw new > > RuntimeException(Intl.getString("object#manager.setupError"), e); > > } > > } > > > > Then, I try to browse into an HDFS system this way: > > String url = String.format("hdfs://%1$s:%2$d/%3$s", > > "hadoop-master ", 50070, hdfsPath); > > return manager.resolveFile(url); > > > > Which results in a bunch of error messages in the log file, which > > looks like it is trying to do user validation on the local machine > > instead of against the Hadoop (remote) cluster. > > Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG > > FileObjectManager: Trying to resolve file reference > > 'hdfs://hadoop-master:50070/' > > Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26) INFO > > org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is > > deprecated. Instead, use fs.defaultFS Apr 11,2014 18:27:39.078 GMT > > T[AWT-EventQueue-0](26) DEBUG > > MutableMetricsFactory: field > > org.apache.hadoop.metrics2.lib.MutableRate > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSucces > > s > > with annotation > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > value=[Rate of successful kerberos logins and latency (milliseconds)], > > about=, type=DEFAULT, always=false, sampleName=Ops) Apr 11,2014 > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > MutableMetricsFactory: field > > org.apache.hadoop.metrics2.lib.MutableRate > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailur > > e > > with annotation > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > value=[Rate of failed kerberos logins and latency (milliseconds)], > > about=, type=DEFAULT, always=false, sampleName=Ops) Apr 11,2014 > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > MutableMetricsFactory: field > > org.apache.hadoop.metrics2.lib.MutableRate > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups > > with annotation > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops) > > Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > MetricsSystemImpl: UgiMetrics, User and group related metrics Apr > > 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups: > > Creating new Groups object > > Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG > > NativeCodeLoader: Trying to load the custom-built native-hadoop > library... > > Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > > NativeCodeLoader: Failed to load native-hadoop with error: > > java.lang.UnsatisfiedLinkError: no hadoop in java.library.path Apr > > 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > > NativeCodeLoader: java.library.path=.... <bunch of stuff> Apr 11,2014 > > 18:27:39.360 GMT T[AWT-EventQueue-0](26) WARN > > NativeCodeLoader: Unable to load native-hadoop library for your > platform... > > using builtin-java classes where applicable Apr 11,2014 18:27:39.375 > > GMT T[AWT-EventQueue-0](26) DEBUG > > JniBasedUnixGroupsMappingWithFallback: Falling back to shell based Apr > > 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG > > JniBasedUnixGroupsMappingWithFallback: Group mapping > > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping > > Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell: > > Failed to detect a valid hadoop home directory: HADOOP_HOME or > > hadoop.home.dir are not set. > > java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. > > at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265) > > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290) > > at > org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) > > at > > org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92) > > at org.apache.hadoop.security.Groups.<init>(Groups.java:76) > > at > > > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:239) > > at > > > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) > > at > > > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) > > at > > > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) > > at > > > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) > > at > > > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) > > at > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473) > > at > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465) > > at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > > at > > > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > > at > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > > at > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > > at > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defa > > ultFileSystemManager.java:609) > > > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell: > > Failed to locate the winutils binary in the hadoop binary path: Could > > not locate executable null\bin\winutils.exe in the Hadoop binaries. > > java.io.IOException: Could not locate executable null\bin\winutils.exe > > in the Hadoop binaries. > > > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups: > > Group mapping > > impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; > > cacheTimeout=300000; warningDeltaMs=5000 Apr 11,2014 18:27:39.469 GMT > > T[AWT-EventQueue-0](26) DEBUG > > UserGroupInformation: hadoop login > > Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG > > UserGroupInformation: hadoop login commit Apr 11,2014 18:27:39.751 GMT > > T[AWT-EventQueue-0](26) DEBUG > > UserGroupInformation: using local user:NTUserPrincipal: <user_name> > > Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG > > UserGroupInformation: UGI loginUser:whiro01 (auth:SIMPLE) Apr 11,2014 > > 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem: > > Error connecting to filesystem hdfs://hadoop-master:50070/: No > > FileSystem for scheme: hdfs > > java.io.IOException: No FileSystem for scheme: hdfs > > at > > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304) > > at > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311) > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) > > at > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) > > at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > > at > > > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > > at > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > > at > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > > at > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defa > > ultFileSystemManager.java:609) > > > > So, my guess is that I don't have enough configuration setup on my > > client machine to tell Hadoop that the authentication is to be done at > > the remote end ....?? So, I'm trying to track down what the > > configuration info might be. > > > > Just hoping there are some Hadoop users/experts here who might be able > > to help. Or maybe I need to talk to Hadoop Users list. Just thought > > since I was using Commons VFS that someone here could help (as in the > > author of the HdfsProvider ....) > > > > Note: I want to build a GUI component that can browse to arbitrary > > HDFS installations, so I can't really be setting up a hard-coded XML > > file for each potential Hadoop cluster I might connect to .... > > > > Thanks, > > ~Roger > > > > -----Original Message----- > > From: Martin Gainty [mailto:[email protected]] > > Sent: Tuesday, April 08, 2014 5:56 PM > > To: Commons Users List > > Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > > access? > > > > Roger- > > > > > > > > look for <artifactId>hadoop-core</artifactId> > > in the <dependencies><dependency> section in your pom.xml > > > > if either > > > > >mvn compile > > > > >mvn package > > cannot locate hadoop-core > > you'll need to sync your <distributionManagement> section in pom.xml > > for > > > > hadoop-core snapshots > > hadoop-core releases > > > > > > Ping back if you're calling no-joy > > > > M- > > > > > > > > > > > From: [email protected] > > > To: [email protected] > > > Subject: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > access? > > > Date: Tue, 8 Apr 2014 21:46:09 +0000 > > > > > > Hi all, > > > I've written a Commons VFS file browser using Apache Pivot (it's > > > part of > > the 2.1 upcoming release). Now I'm trying to use it to browse into > > HDFS files. I got it to work with HDFS 1.x by simple including the > > hadoop-core .jar and a couple of other commons .jars that are also > > needed. But, that approach doesn't work for Hadoop 2.x. Specifically > > the "hadoop-core-xxx.jar" doesn't exist anymore, and substituting the > > "hadoop-hdfs-xxxx.jar" doesn't work at all. So, before I bang my head > > against the wall, does anyone here know what .jars I would need for > > Hadoop 2.x to interface with the VFS HDFS provider? > > > > > > Thanks, > > > ~Roger Whitcomb > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > > -- > E-Mail: [email protected] | [email protected] Java Persistence > with Hibernate, Second Edition<http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- E-Mail: [email protected] | [email protected] Java Persistence with Hibernate, Second Edition<http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
