Hi Ted,
At this point I gave up and implemented the workaround to browse the
NFS mount area instead. The test case was kind of complicated anyway, since it
involves the Pivot VFS File Browser as we as the Commons VFS code.
Thanks,
~Roger
-----Original Message-----
From: Ted Dunning [mailto:[email protected]]
Sent: Tuesday, May 26, 2015 8:22 PM
To: Commons Users List
Subject: Re: [VFS] Could the HDFS connector work with MapR systems?
The NFS mount trick allows all of the files to appear via the local://* file
system.
Otherwise, it is true that the wire protocol for MapR FS is different from
HDFS. In fact, the wire protocol for different versions of HDFS are different
as well.
But what is standard is how different implementations of a Hadoop compatible
file system are plugged into the file space. That should work for any
reasonable application if you have the right jars in your search path.
One possibility where this would break would be if the client code assumes, for
instance, that all file paths must have a server address. Otherwise,
well-behaved HDFS client applications should work. The null host name
exception sounds like there is a format exception going on.
Can I see the some test code and stack traces somewhere?
On Tue, May 26, 2015 at 4:32 PM, Roger Whitcomb <[email protected]>
wrote:
> So, I'm pretty sure (now) that it DOESN'T work. There are comments here:
> http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting
> -to-namenode.html
> and here:
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Application
> s plus the connector here:
> https://github.com/pentaho/pentaho-hdfs-vfs
> that all point to special MapR code being needed, and the standard
> HDFS client protocols not working. Plus all my testing seemed to confirm
> this.
>
> We have figured out a workaround for our configuration, though (we're
> basically requiring an NFS mount of the MapR file system, so we can
> just browse the MapR files as native).
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Gary Gregory [mailto:[email protected]]
> Sent: Tuesday, May 26, 2015 3:48 PM
> To: Commons Users List
> Subject: RE: [VFS] Could the HDFS connector work with MapR systems?
>
> I've not seen VFS used in this configuration before. You could try
> updating to the current HDFS jar files.
> Gary
>
> -------- Original message --------
> From: Roger Whitcomb <[email protected]>
> Date: 05/26/2015 10:39 (GMT-08:00)
> To: Commons Users List <[email protected]>
> Subject: [VFS] Could the HDFS connector work with MapR systems?
>
> Hi all,
> I'm using Commons VFS 2.1 along with the HDFS
> connector and everything is working brilliantly ... until, that is,
> trying to access HDFS on a MapR system. Then I get things like null
> host name exceptions and the like. So, anyone have any idea whether
> the HDFS connector even COULD work against MapR? And if not, anyone
> tried to build a MapR connector to Commons VFS 2.1?
>
> Thanks in advance,
> ~Roger Whitcomb
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
