--Forwarded Message Attachment-- From: [email protected] To: [email protected] Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access? Date: Sat, 12 Apr 2014 08:09:25 -0400
Morning Roger and Gary Brandon Li figured this out last year by installing NFS on Wondows and then mounting a HDFS mountpoint http://lucene.472066.n3.nabble.com/Mounting-HDFS-on-client-td4107671.html Sorry for the long delay in response..Ive been knee deep in multiple house repair items..this has been a long costly winter HTH Martin Gainty ______________________________________________ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. > Date: Fri, 11 Apr 2014 17:49:10 -0400 > Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access? > From: [email protected] > To: [email protected] > > FWIW, I've not had luck getting VFS with Hadoop 1.1.2 to work on Windows. I > run the unit tests on a Linux VM. YMMV. > > Gary > > > On Fri, Apr 11, 2014 at 5:07 PM, Roger Whitcomb > <[email protected]>wrote: > > > My client is on Windows 7, but the servers (Hadoop clusters) are mostly > > Ubuntu (Cloudera VMs) for now. > > > > There doesn't seem to be any problem of Hadoop versions (i.e., it finds > > and loads the Hadoop classes just fine), but it isn't being told what the > > right configuration is. > > > > I've now seem some other postings elsewhere that seem to indicate that the > > FS.defaultFS setting in the Configuration object has to point to the remote > > system. So, I'm guessing that the VFS HdfsProvider needs to know some more > > than I am telling it. Or maybe this is a change from Hadoop 1.x to 2.x?! > > > > Thanks, > > ~Roger > > > > -----Original Message----- > > From: Gary Gregory [mailto:[email protected]] > > Sent: Friday, April 11, 2014 12:58 PM > > To: Commons Users List > > Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > > access? > > > > Curious: What OS are you on? > > > > We build trunk with Hadoop version 1.1.2, so who know what happens with a > > newer version. > > > > I would start with trying to build VFS with the version of Hadoop you want > > to use to see what happens... > > > > Gary > > > > > > On Fri, Apr 11, 2014 at 3:51 PM, Roger Whitcomb > > <[email protected]>wrote: > > > > > Hi Martin, > > > Well, I downloaded Hadoop 2.3.0 from one of the mirrors, and cannot > > > find a "hadoop-core....jar" file anywhere in that distribution. But I > > > was able to track down sort of the minimum set of .jars necessary to > > > at least (try to) connect using Commons VFS 2.1: > > > commons-collections-3.2.1.jar > > > commons-configuration-1.6.jar > > > commons-lang-2.6.jar > > > commons-vfs2-2.1-SNAPSHOT.jar > > > guava-11.0.2.jar > > > hadoop-auth-2.3.0.jar > > > hadoop-common-2.3.0.jar > > > log4j-1.2.17.jar > > > slf4j-api-1.7.5.jar > > > slf4j-log4j12-1.7.5.jar > > > > > > What's happening now is that I instantiated the HdfsProvider this way: > > > private static DefaultFileSystemManager manager = null; > > > > > > static > > > { > > > manager = new DefaultFileSystemManager(); > > > try { > > > manager.setFilesCache(new DefaultFilesCache()); > > > manager.addProvider("hdfs", new HdfsFileProvider()); > > > manager.setFileContentInfoFactory(new > > > FileContentInfoFilenameFactory()); > > > manager.setFilesCache(new SoftRefFilesCache()); > > > manager.setReplicator(new DefaultFileReplicator()); > > > manager.setCacheStrategy(CacheStrategy.ON_RESOLVE); > > > manager.init(); > > > } > > > catch (final FileSystemException e) { > > > throw new > > > RuntimeException(Intl.getString("object#manager.setupError"), e); > > > } > > > } > > > > > > Then, I try to browse into an HDFS system this way: > > > String url = String.format("hdfs://%1$s:%2$d/%3$s", > > > "hadoop-master ", 50070, hdfsPath); > > > return manager.resolveFile(url); > > > > > > Which results in a bunch of error messages in the log file, which > > > looks like it is trying to do user validation on the local machine > > > instead of against the Hadoop (remote) cluster. > > > Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG > > > FileObjectManager: Trying to resolve file reference > > > 'hdfs://hadoop-master:50070/' > > > Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26) INFO > > > org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is > > > deprecated. Instead, use fs.defaultFS Apr 11,2014 18:27:39.078 GMT > > > T[AWT-EventQueue-0](26) DEBUG > > > MutableMetricsFactory: field > > > org.apache.hadoop.metrics2.lib.MutableRate > > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSucces > > > s > > > with annotation > > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > > value=[Rate of successful kerberos logins and latency (milliseconds)], > > > about=, type=DEFAULT, always=false, sampleName=Ops) Apr 11,2014 > > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > > MutableMetricsFactory: field > > > org.apache.hadoop.metrics2.lib.MutableRate > > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailur > > > e > > > with annotation > > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > > value=[Rate of failed kerberos logins and latency (milliseconds)], > > > about=, type=DEFAULT, always=false, sampleName=Ops) Apr 11,2014 > > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > > MutableMetricsFactory: field > > > org.apache.hadoop.metrics2.lib.MutableRate > > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups > > > with annotation > > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, > > > value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops) > > > Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG > > > MetricsSystemImpl: UgiMetrics, User and group related metrics Apr > > > 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups: > > > Creating new Groups object > > > Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG > > > NativeCodeLoader: Trying to load the custom-built native-hadoop > > library... > > > Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > > > NativeCodeLoader: Failed to load native-hadoop with error: > > > java.lang.UnsatisfiedLinkError: no hadoop in java.library.path Apr > > > 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG > > > NativeCodeLoader: java.library.path=.... <bunch of stuff> Apr 11,2014 > > > 18:27:39.360 GMT T[AWT-EventQueue-0](26) WARN > > > NativeCodeLoader: Unable to load native-hadoop library for your > > platform... > > > using builtin-java classes where applicable Apr 11,2014 18:27:39.375 > > > GMT T[AWT-EventQueue-0](26) DEBUG > > > JniBasedUnixGroupsMappingWithFallback: Falling back to shell based Apr > > > 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG > > > JniBasedUnixGroupsMappingWithFallback: Group mapping > > > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping > > > Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell: > > > Failed to detect a valid hadoop home directory: HADOOP_HOME or > > > hadoop.home.dir are not set. > > > java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. > > > at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265) > > > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290) > > > at > > org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) > > > at > > > org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92) > > > at org.apache.hadoop.security.Groups.<init>(Groups.java:76) > > > at > > > > > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:239) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) > > > at > > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473) > > > at > > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465) > > > at > > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331) > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > > > at > > > > > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > > > at > > > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > > > at > > > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > > > at > > > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > > > at > > > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defa > > > ultFileSystemManager.java:609) > > > > > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell: > > > Failed to locate the winutils binary in the hadoop binary path: Could > > > not locate executable null\bin\winutils.exe in the Hadoop binaries. > > > java.io.IOException: Could not locate executable null\bin\winutils.exe > > > in the Hadoop binaries. > > > > > > Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups: > > > Group mapping > > > impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; > > > cacheTimeout=300000; warningDeltaMs=5000 Apr 11,2014 18:27:39.469 GMT > > > T[AWT-EventQueue-0](26) DEBUG > > > UserGroupInformation: hadoop login > > > Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG > > > UserGroupInformation: hadoop login commit Apr 11,2014 18:27:39.751 GMT > > > T[AWT-EventQueue-0](26) DEBUG > > > UserGroupInformation: using local user:NTUserPrincipal: <user_name> > > > Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG > > > UserGroupInformation: UGI loginUser:whiro01 (auth:SIMPLE) Apr 11,2014 > > > 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem: > > > Error connecting to filesystem hdfs://hadoop-master:50070/: No > > > FileSystem for scheme: hdfs > > > java.io.IOException: No FileSystem for scheme: hdfs > > > at > > > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304) > > > at > > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311) > > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90) > > > at > > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350) > > > at > > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332) > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369) > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168) > > > at > > > > > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115) > > > at > > > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84) > > > at > > > > > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64) > > > at > > > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700) > > > at > > > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656) > > > at > > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defa > > > ultFileSystemManager.java:609) > > > > > > So, my guess is that I don't have enough configuration setup on my > > > client machine to tell Hadoop that the authentication is to be done at > > > the remote end ....?? So, I'm trying to track down what the > > > configuration info might be. > > > > > > Just hoping there are some Hadoop users/experts here who might be able > > > to help. Or maybe I need to talk to Hadoop Users list. Just thought > > > since I was using Commons VFS that someone here could help (as in the > > > author of the HdfsProvider ....) > > > > > > Note: I want to build a GUI component that can browse to arbitrary > > > HDFS installations, so I can't really be setting up a hard-coded XML > > > file for each potential Hadoop cluster I might connect to .... > > > > > > Thanks, > > > ~Roger > > > > > > -----Original Message----- > > > From: Martin Gainty [mailto:[email protected]] > > > Sent: Tuesday, April 08, 2014 5:56 PM > > > To: Commons Users List > > > Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > > > access? > > > > > > Roger- > > > > > > > > > > > > look for <artifactId>hadoop-core</artifactId> > > > in the <dependencies><dependency> section in your pom.xml > > > > > > if either > > > > > > >mvn compile > > > > > > >mvn package > > > cannot locate hadoop-core > > > you'll need to sync your <distributionManagement> section in pom.xml > > > for > > > > > > hadoop-core snapshots > > > hadoop-core releases > > > > > > > > > Ping back if you're calling no-joy > > > > > > M- > > > > > > > > > > > > > > > > From: [email protected] > > > > To: [email protected] > > > > Subject: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS > > access? > > > > Date: Tue, 8 Apr 2014 21:46:09 +0000 > > > > > > > > Hi all, > > > > I've written a Commons VFS file browser using Apache Pivot (it's > > > > part of > > > the 2.1 upcoming release). Now I'm trying to use it to browse into > > > HDFS files. I got it to work with HDFS 1.x by simple including the > > > hadoop-core .jar and a couple of other commons .jars that are also > > > needed. But, that approach doesn't work for Hadoop 2.x. Specifically > > > the "hadoop-core-xxx.jar" doesn't exist anymore, and substituting the > > > "hadoop-hdfs-xxxx.jar" doesn't work at all. So, before I bang my head > > > against the wall, does anyone here know what .jars I would need for > > > Hadoop 2.x to interface with the VFS HDFS provider? > > > > > > > > Thanks, > > > > ~Roger Whitcomb > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > -- > > E-Mail: [email protected] | [email protected] Java Persistence > > with Hibernate, Second Edition<http://www.manning.com/bauer3/> > > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > > Spring Batch in Action <http://www.manning.com/templier/> > > Blog: http://garygregory.wordpress.com > > Home: http://garygregory.com/ > > Tweet! http://twitter.com/GaryGregory > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > > -- > E-Mail: [email protected] | [email protected] > Java Persistence with Hibernate, Second > Edition<http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory
