Adam - I am new to using Jira properly. (I couldn't find the JIRA for the Tachyon change as an example, so I linked to the code... is that ok?)
I created https://issues.apache.org/jira/browse/MESOS-1711 If you wouldn't mind taking a quick look to make sure I filled things out correctly to get addressed I'd appreciate it. If you want to hit me up off list with any recommendations on what I did to make it better in the future, I'd appreciate it as well. Thanks! John On Mon, Aug 18, 2014 at 4:43 AM, Adam Bordelon <[email protected]> wrote: > Okay, I guess MapRFS is protocol compatible with HDFS, but not > uri-compatible. I know the MapR guys have gotten MapR on Mesos working. > They may have more answers for you on how they accomplished this. > > > why hard code the file prefixes? > We allow any uri, so we need to have handlers coded for each type of > protocol group, which so far includes hdfs/hftp/s3/s3n which use > hdfs::copyToLocal, or http/https/ftp/ftps which use net::download, or > file:// or an absolute/relative path for files pre-populated on the machine > (uses 'cp'). MapRFS (and Tachyon) would probably fit into the > hdfs::copyToLocal group so easily that it would be a one-line fix each. > > > I really think the hdfs vs other prefixes should be looked at > I agree. Could you file a JIRA with your request? It should be an easy > enough change for us to pick up. I would also like to see Tachyon as a > possible filesystem for the fetcher. > > > On Fri, Aug 15, 2014 at 5:16 PM, John Omernik <[email protected]> wrote: > >> I tried hdfs:/// and hdfs://cldbnode:7222/ Neither worked (examples >> below) I really think the hdfs vs other prefixes should be looked at. Like >> I said above, the tachyon project just added a env variable to address >> this. >> >> >> >> hdfs://cldbnode:7222/ >> >> WARNING: Logging before InitGoogleLogging() is written to STDERR >> I0815 19:14:17.101666 22022 fetcher.cpp:76] Fetching URI >> 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> I0815 19:14:17.101780 22022 fetcher.cpp:105] Downloading resource from >> 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' >> E0815 19:14:17.778833 22022 fetcher.cpp:109] HDFS copyToLocal failed: hadoop >> fs -copyToLocal 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' >> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please >> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties >> files. >> -copyToLocal: Wrong FS: >> maprfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: >> hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] >> <src> ... <localdst> >> Failed to fetch: hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Failed to synchronize with slave (it's probably exited) >> >> >> >> >> hdfs:/// >> >> >> >> >> >> I0815 19:10:45.006803 21508 fetcher.cpp:76] Fetching URI >> 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> I0815 19:10:45.007099 21508 fetcher.cpp:105] Downloading resource from >> 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' >> E0815 19:10:45.681922 21508 fetcher.cpp:109] HDFS copyToLocal failed: hadoop >> fs -copyToLocal 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >> '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' >> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please >> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties >> files. >> -copyToLocal: Wrong FS: maprfs:/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, >> expected: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] [-crc] >> <src> ... <localdst> >> Failed to fetch: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >> Failed to synchronize with slave (it's probably exited) >> >> >> >> On Fri, Aug 15, 2014 at 5:38 PM, John Omernik <[email protected]> wrote: >> >>> I am away from my cluster right now, I trued doing a hadoop fs -ls >>> maprfs:// and that worked. When I tries hadoop fs -ls hdfs:/// it failed >>> with wrong fs type. With that error I didn't try it in the mapred-site. I >>> will try it. Still...why hard code the file prefixes? I guess I am curious >>> on how glusterfs would work, or others as they pop up. >>> On Aug 15, 2014 5:04 PM, "Adam Bordelon" <[email protected]> wrote: >>> >>>> Can't you just use the hdfs:// protocol for maprfs? That should work >>>> just fine. >>>> >>>> >>>> On Fri, Aug 15, 2014 at 2:50 PM, John Omernik <[email protected]> wrote: >>>> >>>>> Thanks all. >>>>> >>>>> I realized MapR has a work around for me that I will try soon in that >>>>> I have MapR fs NFS mounted on each node, I.e. I should be able to get the >>>>> tar from there. >>>>> >>>>> That said, perhaps someone with better coding skills than me could >>>>> provide an env variable where a user could provide the HDFS prefixes to >>>>> try. I know we did that with the tachyon project and it works well for >>>>> other HDFS compatible fs implementations, perhaps that would work here? >>>>> Hard coding a pluggable system seems like a long term issue that will keep >>>>> coming up. >>>>> On Aug 15, 2014 4:02 PM, "Tim St Clair" <[email protected]> wrote: >>>>> >>>>>> The uri doesn't currently start with any of the known types (at least >>>>>> on 1st grok). >>>>>> You could redirect via a proxy that does the job for you. >>>>>> >>>>>> | if you had some fuse mount that would work too. >>>>>> >>>>>> Cheers, >>>>>> Tim >>>>>> >>>>>> ------------------------------ >>>>>> >>>>>> *From: *"John Omernik" <[email protected]> >>>>>> *To: *[email protected] >>>>>> *Sent: *Friday, August 15, 2014 3:55:02 PM >>>>>> *Subject: *Alternate HDFS Filesystems + Hadoop on Mesos >>>>>> >>>>>> I am on a wonderful journey trying to get hadoop on Mesos working >>>>>> with MapR. I feel like I am close, but when the slaves try to run the >>>>>> packaged Hadoop, I get the error below. The odd thing is, I KNOW I got >>>>>> Spark running on Mesos pulling both data and the packages from MapRFS. >>>>>> So >>>>>> I am confused why there is and issue with the fetcher.cpp here. Granted, >>>>>> when I got spark working, it was on 0.19.0, and I am trying a "fresh" >>>>>> version from git (0.20.0?) that I just pulled today. I am not sure if >>>>>> that >>>>>> work, but when I have more time I will try spark again. >>>>>> >>>>>> Any thoughts on this error? Thanks. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Error: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>>>>> I0815 15:48:35.446071 20636 fetcher.cpp:76] Fetching URI >>>>>> 'maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' >>>>>> E0815 15:48:35.446184 20636 fetcher.cpp:161] A relative path was passed >>>>>> for the resource but the environment variable MESOS_FRAMEWORKS_HOME is >>>>>> not set. Please either specify this config option or avoid using a >>>>>> relative path >>>>>> Failed to fetch: maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz >>>>>> Failed to synchronize with slave (it's probably exited) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> Timothy St. Clair >>>>>> Red Hat Inc. >>>>>> >>>>> >>>> >> >
