Conner - Awesome!
+1 re: java + PR to tachyon repo I should have some time later this week to eval, I'm curious how you handled the mounting details. Cheers, Tim ----- Original Message ----- > From: "Connor Doyle" <[email protected]> > To: [email protected] > Cc: "Haoyuan Li" <[email protected]>, "Huamin Chen" <[email protected]>, > "Brad Childs" <[email protected]>, "Adam Bordelon" <[email protected]> > Sent: Sunday, August 24, 2014 6:46:46 PM > Subject: Re: Alternate HDFS Filesystems + Hadoop on Mesos > > Also, fwiw I'm interested in rallying folks on a Tachyon Framework in the > > not-too-distant future, for anyone who is interested. Probably follow the > > spark model and try to push upstream. > > Hi Tim, late follow-up: > The not-too distant future is here! Adam and I took a stab at a Tachyon > framework during the MesosCon hackathon ( > http://github.com/mesosphere/tachyon-mesos ). > We started writing in Scala, but not at all opposed to switching to Java, > especially if the work can be upstreamed. > -- > Connor > > > On Fri, Aug 15, 2014 at 5:16 PM, John Omernik < [email protected] > wrote: > > > > > > > I tried hdfs:/// and hdfs://cldbnode:7222/ Neither worked (examples > > > > below) > > > > I > > > > really think the hdfs vs other prefixes should be looked at. Like I > > > > said > > > > above, the tachyon project just added a env variable to address this. > > > > > > > > > > hdfs://cldbnode:7222/ > > > > > > > > > > WARNING: Logging before InitGoogleLogging() is written to STDERR > > > > > > > > > > I0815 19:14:17.101666 22022 fetcher.cpp:76] Fetching URI > > > > 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > I0815 19:14:17.101780 22022 fetcher.cpp:105] Downloading resource from > > > > 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to > > > > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > E0815 19:14:17.778833 22022 fetcher.cpp:109] HDFS copyToLocal failed: > > > > hadoop > > > > fs -copyToLocal > > > > 'hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0003/executors/executor_Task_Tracker_5/runs/b3174e72-75ea-48be-bbb8-a9a6cc605018/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. > > > > Please > > > > use > > > > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties > > > > files. > > > > > > > > > > -copyToLocal: Wrong FS: > > > > maprfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, expected: > > > > hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz > > > > > > > > > > Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] > > > > [-crc] > > > > <src> ... <localdst> > > > > > > > > > > Failed to fetch: > > > > hdfs://hadoopmapr1:7222/mesos/hadoop-0.20.2-mapr-4.0.0.tgz > > > > > > > > > > Failed to synchronize with slave (it's probably exited) > > > > > > > > > > hdfs:/// > > > > > > > > > > I0815 19:10:45.006803 21508 fetcher.cpp:76] Fetching URI > > > > 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > I0815 19:10:45.007099 21508 fetcher.cpp:105] Downloading resource from > > > > 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' to > > > > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > E0815 19:10:45.681922 21508 fetcher.cpp:109] HDFS copyToLocal failed: > > > > hadoop > > > > fs -copyToLocal 'hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > '/tmp/mesos/slaves/20140815-103603-1677764800-5050-24315-2/frameworks/20140815-154511-1677764800-5050-7162-0002/executors/executor_Task_Tracker_2/runs/22689054-aff6-4f7c-9746-a068a11ff000/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. > > > > Please > > > > use > > > > org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties > > > > files. > > > > > > > > > > -copyToLocal: Wrong FS: maprfs:/mesos/hadoop-0.20.2-mapr-4.0.0.tgz, > > > > expected: > > > > hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz > > > > > > > > > > Usage: hadoop fs [generic options] -copyToLocal [-p] [-ignoreCrc] > > > > [-crc] > > > > <src> ... <localdst> > > > > > > > > > > Failed to fetch: hdfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz > > > > > > > > > > Failed to synchronize with slave (it's probably exited) > > > > > > > > > > On Fri, Aug 15, 2014 at 5:38 PM, John Omernik < [email protected] > > > > > wrote: > > > > > > > > > > > I am away from my cluster right now, I trued doing a hadoop fs -ls > > > > > maprfs:// > > > > > and that worked. When I tries hadoop fs -ls hdfs:/// it failed with > > > > > wrong > > > > > fs > > > > > type. With that error I didn't try it in the mapred-site. I will try > > > > > it. > > > > > Still...why hard code the file prefixes? I guess I am curious on how > > > > > glusterfs would work, or others as they pop up. > > > > > > > > > > > > > > > On Aug 15, 2014 5:04 PM, "Adam Bordelon" < [email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > Can't you just use the hdfs:// protocol for maprfs? That should > > > > > > work > > > > > > just > > > > > > fine. > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 15, 2014 at 2:50 PM, John Omernik < [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Thanks all. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I realized MapR has a work around for me that I will try soon in > > > > > > > that > > > > > > > I > > > > > > > have > > > > > > > MapR fs NFS mounted on each node, I.e. I should be able to get > > > > > > > the > > > > > > > tar > > > > > > > from > > > > > > > there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > That said, perhaps someone with better coding skills than me > > > > > > > could > > > > > > > provide > > > > > > > an > > > > > > > env variable where a user could provide the HDFS prefixes to try. > > > > > > > I > > > > > > > know > > > > > > > we > > > > > > > did that with the tachyon project and it works well for other > > > > > > > HDFS > > > > > > > compatible fs implementations, perhaps that would work here? Hard > > > > > > > coding > > > > > > > a > > > > > > > pluggable system seems like a long term issue that will keep > > > > > > > coming > > > > > > > up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Aug 15, 2014 4:02 PM, "Tim St Clair" < [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The uri doesn't currently start with any of the known types (at > > > > > > > > least > > > > > > > > on > > > > > > > > 1st > > > > > > > > grok). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You could redirect via a proxy that does the job for you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > | if you had some fuse mount that would work too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tim > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: "John Omernik" < [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent: Friday, August 15, 2014 3:55:02 PM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: Alternate HDFS Filesystems + Hadoop on Mesos > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am on a wonderful journey trying to get hadoop on Mesos > > > > > > > > > working > > > > > > > > > with > > > > > > > > > MapR. > > > > > > > > > I feel like I am close, but when the slaves try to run the > > > > > > > > > packaged > > > > > > > > > Hadoop, > > > > > > > > > I get the error below. The odd thing is, I KNOW I got Spark > > > > > > > > > running > > > > > > > > > on > > > > > > > > > Mesos > > > > > > > > > pulling both data and the packages from MapRFS. So I am > > > > > > > > > confused > > > > > > > > > why > > > > > > > > > there > > > > > > > > > is and issue with the fetcher.cpp here. Granted, when I got > > > > > > > > > spark > > > > > > > > > working, > > > > > > > > > it was on 0.19.0, and I am trying a "fresh" version from git > > > > > > > > > (0.20.0?) > > > > > > > > > that > > > > > > > > > I just pulled today. I am not sure if that work, but when I > > > > > > > > > have > > > > > > > > > more > > > > > > > > > time > > > > > > > > > I > > > > > > > > > will try spark again. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any thoughts on this error? Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Error: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WARNING: Logging before InitGoogleLogging() is written to > > > > > > > > > STDERR > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I0815 15:48:35.446071 20636 fetcher.cpp:76] Fetching URI > > > > > > > > > 'maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz' > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > E0815 15:48:35.446184 20636 fetcher.cpp:161] A relative path > > > > > > > > > was > > > > > > > > > passed > > > > > > > > > for > > > > > > > > > the resource but the environment variable > > > > > > > > > MESOS_FRAMEWORKS_HOME > > > > > > > > > is > > > > > > > > > not > > > > > > > > > set. > > > > > > > > > Please either specify this config option or avoid using a > > > > > > > > > relative > > > > > > > > > path > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Failed to fetch: maprfs:///mesos/hadoop-0.20.2-mapr-4.0.0.tgz > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Failed to synchronize with slave (it's probably exited) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Timothy St. Clair > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Red Hat Inc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Cheers, > > > Timothy St. Clair > > > Red Hat Inc. > -- Cheers, Timothy St. Clair Red Hat Inc.
