Thanks a lot John, the JIRA looks great! We'll get on it soon. Tim, I'd be happy to shepherd this change for you. Also, we spoke with H.Y. from Tachyon at Mesosphere recently, and we (I and probably others) would be glad to help with the Tachyon Framework as well. Should be much easier than handling persistent storage like HDFS.
On Mon, Aug 18, 2014 at 8:08 AM, Tim St Clair <[email protected]> wrote: > inline > > ------------------------------ > > *From: *"Adam Bordelon" <[email protected]> > *To: *[email protected] > *Sent: *Monday, August 18, 2014 4:43:34 AM > *Subject: *Re: Alternate HDFS Filesystems + Hadoop on Mesos > > > Okay, I guess MapRFS is protocol compatible with HDFS, but not > uri-compatible. I know the MapR guys have gotten MapR on Mesos working. > They may have more answers for you on how they accomplished this. > > > why hard code the file prefixes? > We allow any uri, so we need to have handlers coded for each type of > protocol group, which so far includes hdfs/hftp/s3/s3n which use > hdfs::copyToLocal, or http/https/ftp/ftps which use net::download, or > file:// or an absolute/relative path for files pre-populated on the machine > (uses 'cp'). MapRFS (and Tachyon) would probably fit into the > hdfs::copyToLocal group so easily that it would be a one-line fix each. > > > I really think the hdfs vs other prefixes should be looked at > I agree. Could you file a JIRA with your request? It should be an easy > enough change for us to pick up. I would also like to see Tachyon as a > possible filesystem for the fetcher. > > +1 & I'll take. IMHO we should check support for *HCFS, which should be a > pretty straight forward check. > > Also, fwiw I'm interested in rallying folks on a Tachyon Framework in the > not-too-distant future, for anyone who is interested. Probably follow the > spark model and try to push upstream. > > > > On Fri, Aug 15, 2014 at 5:16 PM, John Omernik <[email protected]> wrote: > >> I tried hdfs:/// and hdfs://cldbnode:7222/ Neither worked (examples >> below) I really think the hdfs vs other prefixes should be looked at. Like >> I said above, the tachyon project just added a env variable to address >> this. >> >> >> >> hdfs://cldbnode:7222/ >> >> WARNING: Logging before InitGoogleLogging() is written to STDERR >> I0815 19:14:17.101666 22022 fetcher.cpp:76] Fetching URI >> 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> I0815 19:14:17.101780 22022 fetcher.cpp:105] Downloading resource from >> 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' >> E0815 19:14:17.778833 22022 fetcher.cpp:109] HDFS copyToLocal failed: hadoop >> fs -copyToLocal 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' >> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please >> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties >> files. >> -copyToLocal: Wrong FS: >> maprfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: >> hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] >> <src> ... <localdst> >> Failed to fetch: hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Failed to synchronize with slave (it's probably exited) >> >> >> >> >> hdfs:/// >> >> >> >> >> I0815 19:10:45.006803 21508 fetcher.cpp:76] Fetching URI >> 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> I0815 19:10:45.007099 21508 fetcher.cpp:105] Downloading resource from >> 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' >> E0815 19:10:45.681922 21508 fetcher.cpp:109] HDFS copyToLocal failed: hadoop >> fs -copyToLocal 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' >> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please >> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties >> files. >> -copyToLocal: Wrong FS: maprfs:/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, >> expected: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] >> <src> ... <localdst> >> Failed to fetch: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Failed to synchronize with slave (it's probably exited) >> >> >> >> On Fri, Aug 15, 2014 at 5:38 PM, John Omernik <[email protected]> wrote: >> >>> I am away from my cluster right now, I trued doing a hadoop fs -ls >>> maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed >>> with wrong fs type. With that error I didn't try it in the mapred-site. I >>> will try it. Still...why hard code the file prefixes? I guess I am curious >>> on how glusterfs would work, or others as they pop up. >>> On Aug 15, 2014 5:04 PM, "Adam Bordelon" <[email protected]> wrote: >>> >>>> Can't you just use the hdfs:// protocol for maprfs? That should work >>>> just fine. >>>> >>>> >>>> On Fri, Aug 15, 2014 at 2:50 PM, John Omernik <[email protected]> wrote: >>>> >>>>> Thanks all. >>>>> >>>>> I realized MapR has a work around for me that I will try soon in that >>>>> I have MapR fs NFS mounted on each node, I.e. I should be able to get the >>>>> tar from there. >>>>> >>>>> That said, perhaps someone with better coding skills than me could >>>>> provide an env variable where a user could provide the HDFS prefixes to >>>>> try. I know we did that with the tachyon project and it works well for >>>>> other HDFS compatible fs implementations, perhaps that would work here? >>>>> Hard coding a pluggable system seems like a long term issue that will keep >>>>> coming up. >>>>> On Aug 15, 2014 4:02 PM, "Tim St Clair" <[email protected]> wrote: >>>>> >>>>>> The uri doesn't currently start with any of the known types (at least >>>>>> on 1st grok). >>>>>> You could redirect via a proxy that does the job for you. >>>>>> >>>>>> | if you had some fuse mount that would work too. >>>>>> >>>>>> Cheers, >>>>>> Tim >>>>>> >>>>>> ------------------------------ >>>>>> >>>>>> *From: *"John Omernik" <[email protected]> >>>>>> *To: *[email protected] >>>>>> *Sent: *Friday, August 15, 2014 3:55:02 PM >>>>>> *Subject: *Alternate HDFS Filesystems + Hadoop on Mesos >>>>>> >>>>>> I am on a wonderful journey trying to get hadoop on Mesos working >>>>>> with MapR. I feel like I am close, but when the slaves try to run the >>>>>> packaged Hadoop, I get the error below. The odd thing is, I KNOW I got >>>>>> Spark running on Mesos pulling both data and the packages from MapRFS. >>>>>> So >>>>>> I am confused why there is and issue with the fetcher.cpp here. Granted, >>>>>> when I got spark working, it was on 0.19.0, and I am trying a "fresh" >>>>>> version from git (0.20.0?) that I just pulled today. I am not sure if >>>>>> that >>>>>> work, but when I have more time I will try spark again. >>>>>> >>>>>> Any thoughts on this error? Thanks. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Error: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>>>>> I0815 15:48:35.446071 20636 fetcher.cpp:76] Fetching URI >>>>>> 'maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >>>>>> E0815 15:48:35.446184 20636 fetcher.cpp:161] A relative path was passed >>>>>> for the resource but the environment variable MESOS_FRAMEWORKS_HOME is >>>>>> not set. Please either specify this config option or avoid using a >>>>>> relative path >>>>>> Failed to fetch: maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >>>>>> Failed to synchronize with slave (it's probably exited) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> Timothy St. Clair >>>>>> Red Hat Inc. >>>>>> >>>>> >>>> >> > > > > -- > Cheers, > Timothy St. Clair > Red Hat Inc. >
