Curious: What OS are you on? We build trunk with Hadoop version 1.1.2, so who know what happens with a newer version.
I would start with trying to build VFS with the version of Hadoop you want to use to see what happens... Gary On Fri, Apr 11, 2014 at 3:51 PM, Roger Whitcomb <[email protected]>wrote: > Hi Martin, > Well, I downloaded Hadoop 2.3.0 from one of the mirrors, and cannot find a > "hadoop-core....jar" file anywhere in that distribution. But I was able to > track down sort of the minimum set of .jars necessary to at least (try to) > connect using Commons VFS 2.1: > commons-collections-3.2.1.jar > commons-configuration-1.6.jar > commons-lang-2.6.jar > commons-vfs2-2.1-SNAPSHOT.jar > guava-11.0.2.jar > hadoop-auth-2.3.0.jar > hadoop-common-2.3.0.jar > log4j-1.2.17.jar > slf4j-api-1.7.5.jar > slf4j-log4j12-1.7.5.jar > > What's happening now is that I instantiated the HdfsProvider this way: > private static DefaultFileSystemManager manager = null; > > static > { > manager = new DefaultFileSystemManager(); > try { > manager.setFilesCache(new DefaultFilesCache()); > manager.addProvider("hdfs", new HdfsFileProvider()); > manager.setFileContentInfoFactory(new > FileContentInfoFilenameFactory()); > manager.setFilesCache(new SoftRefFilesCache()); > manager.setReplicator(new DefaultFileReplicator()); > manager.setCacheStrategy(CacheStrategy.ON_RESOLVE); > manager.init(); > } > catch (final FileSystemException e) { > throw new > RuntimeException(Intl.getString("object#manager.setupError"), e); > } > } > > Then, I try to browse into an HDFS system this way: > String url = String.format("hdfs://%1$s:%2$d/%3$s", > "hadoop-master ", 50070, hdfsPath); > return manager.resolveFile(url); > > Which results in a bunch of error messages in the log file, which looks > like it is trying to do user validation on the local machine instead of > against the Hadoop (remote) cluster. > Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG > FileObjectManager: Trying to resolve file reference > 'hdfs://hadoop-master:50070/' > Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26) INFO > org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is > deprecated. Instead, use fs.defaultFS > Apr 11,2014 18:27:39.078 GMT T[AWT-EventQueue-0](26) DEBUG > MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess > with annotation > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate > of successful kerberos logins and latency (milliseconds)], about=, > type=DEFAULT, always=false, sampleName=Ops) > Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure > with annotation > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate > of failed kerberos logins and latency (milliseconds)], about=, > type=DEFAULT, always=false, sampleName=Ops) > Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with > annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops) > Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > MetricsSystemImpl: UgiMetrics, User and group related metrics > Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups: > Creating new Groups object > Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG > NativeCodeLoader: Trying to load the custom-built native-hadoop library... > Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > NativeCodeLoader: Failed to load native-hadoop with error: > java.lang.UnsatisfiedLinkError: no hadoop in java.library.path > Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > NativeCodeLoader: java.library.path=.... <bunch of stuff> > Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) WARN > NativeCodeLoader: Unable to load native-hadoop library for your platform... > using builtin-java classes where applicable > Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG > JniBasedUnixGroupsMappingWithFallback: Falling back to shell based > Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG > JniBasedUnixGroupsMappingWithFallback: Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping > Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed > to detect a valid hadoop home directory: HADOOP_HOME or hadoop.home.dir are > not set. > java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. > at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265) > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290) > at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) > at > org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92) > at org.apache.hadoop.security.Groups.<init>(Groups.java:76) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:239) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) > at > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) > at > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) > at > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473) > at > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > at > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:609) > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed > to locate the winutils binary in the hadoop binary path: Could not locate > executable null\bin\winutils.exe in the Hadoop binaries. > java.io.IOException: Could not locate executable null\bin\winutils.exe in > the Hadoop binaries. > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups: Group > mapping > impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; > cacheTimeout=300000; warningDeltaMs=5000 > Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG > UserGroupInformation: hadoop login > Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG > UserGroupInformation: hadoop login commit > Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG > UserGroupInformation: using local user:NTUserPrincipal: <user_name> > Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG > UserGroupInformation: UGI loginUser:whiro01 (auth:SIMPLE) > Apr 11,2014 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem: > Error connecting to filesystem hdfs://hadoop-master:50070/: No FileSystem > for scheme: hdfs > java.io.IOException: No FileSystem for scheme: hdfs > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > at > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > at > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > at > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:609) > > So, my guess is that I don't have enough configuration setup on my client > machine to tell Hadoop that the authentication is to be done at the remote > end ....?? So, I'm trying to track down what the configuration info might > be. > > Just hoping there are some Hadoop users/experts here who might be able to > help. Or maybe I need to talk to Hadoop Users list. Just thought since I > was using Commons VFS that someone here could help (as in the author of the > HdfsProvider ....) > > Note: I want to build a GUI component that can browse to arbitrary HDFS > installations, so I can't really be setting up a hard-coded XML file for > each potential Hadoop cluster I might connect to .... > > Thanks, > ~Roger > > -----Original Message----- > From: Martin Gainty [mailto:[email protected]] > Sent: Tuesday, April 08, 2014 5:56 PM > To: Commons Users List > Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > access? > > Roger- > > > > look for <artifactId>hadoop-core</artifactId> > in the <dependencies><dependency> section in your pom.xml > > if either > > >mvn compile > > >mvn package > cannot locate hadoop-core > you'll need to sync your <distributionManagement> section in pom.xml for > > hadoop-core snapshots > hadoop-core releases > > > Ping back if you're calling no-joy > > M- > > > > > > From: [email protected] > > To: [email protected] > > Subject: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access? > > Date: Tue, 8 Apr 2014 21:46:09 +0000 > > > > Hi all, > > I've written a Commons VFS file browser using Apache Pivot (it's part of > the 2.1 upcoming release). Now I'm trying to use it to browse into HDFS > files. I got it to work with HDFS 1.x by simple including the hadoop-core > .jar and a couple of other commons .jars that are also needed. But, that > approach doesn't work for Hadoop 2.x. Specifically the > "hadoop-core-xxx.jar" doesn't exist anymore, and substituting the > "hadoop-hdfs-xxxx.jar" doesn't work at all. So, before I bang my head > against the wall, does anyone here know what .jars I would need for Hadoop > 2.x to interface with the VFS HDFS provider? > > > > Thanks, > > ~Roger Whitcomb > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- E-Mail: [email protected] | [email protected] Java Persistence with Hibernate, Second Edition<http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
