Okay, I guess MapRFS is protocol compatible with HDFS, but not uri-compatible. I know the MapR guys have gotten MapR on Mesos working. They may have more answers for you on how they accomplished this.
> why hard code the file prefixes? We allow any uri, so we need to have handlers coded for each type of protocol group, which so far includes hdfs/hftp/s3/s3n which use hdfs::copyToLocal, or http/https/ftp/ftps which use net::download, or file:// or an absolute/relative path for files pre-populated on the machine (uses 'cp'). MapRFS (and Tachyon) would probably fit into the hdfs::copyToLocal group so easily that it would be a one-line fix each. > I really think the hdfs vs other prefixes should be looked at I agree. Could you file a JIRA with your request? It should be an easy enough change for us to pick up. I would also like to see Tachyon as a possible filesystem for the fetcher. On Fri, Aug 15, 2014 at 5:16 PM, John Omernik <[email protected]> wrote: > I tried hdfs:/// and hdfs://cldbnode:7222/ Neither worked (examples below) > I really think the hdfs vs other prefixes should be looked at. Like I said > above, the tachyon project just added a env variable to address this. > > > > hdfs://cldbnode:7222/ > > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0815 19:14:17.101666 22022 fetcher.cpp:76] Fetching URI > 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > I0815 19:14:17.101780 22022 fetcher.cpp:105] Downloading resource from > 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' > E0815 19:14:17.778833 22022 fetcher.cpp:109] HDFS copyToLocal failed: hadoop > fs -copyToLocal 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. > -copyToLocal: Wrong FS: > maprfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: > hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz > Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] > <src> ... <localdst> > Failed to fetch: hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz > Failed to synchronize with slave (it's probably exited) > > > > > hdfs:/// > > > > > I0815 19:10:45.006803 21508 fetcher.cpp:76] Fetching URI > 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > I0815 19:10:45.007099 21508 fetcher.cpp:105] Downloading resource from > 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' > E0815 19:10:45.681922 21508 fetcher.cpp:109] HDFS copyToLocal failed: hadoop > fs -copyToLocal 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. > -copyToLocal: Wrong FS: maprfs:/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: > hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz > Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] > <src> ... <localdst> > Failed to fetch: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz > Failed to synchronize with slave (it's probably exited) > > > > On Fri, Aug 15, 2014 at 5:38 PM, John Omernik <[email protected]> wrote: > >> I am away from my cluster right now, I trued doing a hadoop fs -ls >> maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed >> with wrong fs type. With that error I didn't try it in the mapred-site. I >> will try it. Still...why hard code the file prefixes? I guess I am curious >> on how glusterfs would work, or others as they pop up. >> On Aug 15, 2014 5:04 PM, "Adam Bordelon" <[email protected]> wrote: >> >>> Can't you just use the hdfs:// protocol for maprfs? That should work >>> just fine. >>> >>> >>> On Fri, Aug 15, 2014 at 2:50 PM, John Omernik <[email protected]> wrote: >>> >>>> Thanks all. >>>> >>>> I realized MapR has a work around for me that I will try soon in that I >>>> have MapR fs NFS mounted on each node, I.e. I should be able to get the tar >>>> from there. >>>> >>>> That said, perhaps someone with better coding skills than me could >>>> provide an env variable where a user could provide the HDFS prefixes to >>>> try. I know we did that with the tachyon project and it works well for >>>> other HDFS compatible fs implementations, perhaps that would work here? >>>> Hard coding a pluggable system seems like a long term issue that will keep >>>> coming up. >>>> On Aug 15, 2014 4:02 PM, "Tim St Clair" <[email protected]> wrote: >>>> >>>>> The uri doesn't currently start with any of the known types (at least >>>>> on 1st grok). >>>>> You could redirect via a proxy that does the job for you. >>>>> >>>>> | if you had some fuse mount that would work too. >>>>> >>>>> Cheers, >>>>> Tim >>>>> >>>>> ------------------------------ >>>>> >>>>> *From: *"John Omernik" <[email protected]> >>>>> *To: *[email protected] >>>>> *Sent: *Friday, August 15, 2014 3:55:02 PM >>>>> *Subject: *Alternate HDFS Filesystems + Hadoop on Mesos >>>>> >>>>> I am on a wonderful journey trying to get hadoop on Mesos working with >>>>> MapR. I feel like I am close, but when the slaves try to run the >>>>> packaged >>>>> Hadoop, I get the error below. The odd thing is, I KNOW I got Spark >>>>> running on Mesos pulling both data and the packages from MapRFS. So I am >>>>> confused why there is and issue with the fetcher.cpp here. Granted, when I >>>>> got spark working, it was on 0.19.0, and I am trying a "fresh" version >>>>> from >>>>> git (0.20.0?) that I just pulled today. I am not sure if that work, but >>>>> when I have more time I will try spark again. >>>>> >>>>> Any thoughts on this error? Thanks. >>>>> >>>>> >>>>> >>>>> >>>>> Error: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>>>> I0815 15:48:35.446071 20636 fetcher.cpp:76] Fetching URI >>>>> 'maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >>>>> E0815 15:48:35.446184 20636 fetcher.cpp:161] A relative path was passed >>>>> for the resource but the environment variable MESOS_FRAMEWORKS_HOME is >>>>> not set. Please either specify this config option or avoid using a >>>>> relative path >>>>> Failed to fetch: maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >>>>> Failed to synchronize with slave (it's probably exited) >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> Timothy St. Clair >>>>> Red Hat Inc. >>>>> >>>> >>> >
