Thanks Bernardo, I will update it on https://github.com/apache/spark/blob/71d4ed271bcbddb154643bd44297ed77190e75cf/project/SparkBuild.scala
in our fork. On Tue, Mar 25, 2014 at 9:49 PM, Bernardo Gomez Palacio < [email protected]> wrote: > I built Spark for hadoop 2.2 with Mesos, make sure that the YARN env var is > disabled. I updated the Mesos dep to point to a Mesos version that uses > protobufs 2.5.0. That should work with CDH5. > > > > On Tuesday, March 25, 2014, Gary Malouf <[email protected]> wrote: > > > Did you build the spark hadoop 2.2 yarn version or just the regular one? > > There is no CDH5 build available. We used spark 0.8.1 built with > > cdh4.2.1-mrv1 before. > > > > We will give it a shot tomorrow and see what happens :) > > On Mar 25, 2014 9:01 PM, "Bernardo Gomez Palacio" < > > [email protected] <javascript:;>> wrote: > > > > > In that setup I am running Spark on Mesos not YARN. > > > > > > On Tue, Mar 25, 2014 at 5:55 PM, Gary Malouf <[email protected] > <javascript:;> > > > > > > wrote: > > > > Thank you for the feedback. Are you running Yarn as a part of this > > > setup? > > > > Seems like that would collide with Mesos. We have a job or 2 written > > in > > > > MRV1 that from what I understand can still be run with 2.2 somehow... > > > > On Mar 25, 2014 8:49 PM, "Bernardo Gomez Palacio" < > > > > [email protected] <javascript:;>> wrote: > > > > > > > >> Hi Gary. > > > >> I am currently using > > > >> Mesos 0.16.0 build for Protobufs 2.5.0 > > > >> (https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x) > > > >> Hadoop version 2.2 > > > >> Spark 0.9.0 ( > > > >> > > https://github.com/guavus/incubator-spark/tree/guavus/branch-0.9-hdp2.2 > > > ) > > > >> > > > >> Things to know. > > > >> Hadoop 1.0.4 uses protobufs 2.4.1 > > > >> Hadoop 2.2 and 2.3 uses protobufs 2.5.0 > > > >> Mesos < 0.17.0 uses protobufs 2.4.1 > > > >> Mesos >= 0.17.0 uses protobufs 2.5.0 > > > >> > > > >> If you need Mesos 0.15.0 on protobufs 2.5.0 you could try to > backport > > > >> https://github.com/Guavus/mesos/tree/releases/0.16.0-HDP2x but if > you > > > >> keep CDH4 with protobufs 2.4.1 you will run into trouble. > > > >> > > > >> Let me know if this help. > > > >> Bernardo. > > > >> > > > >> > > > >> > > > >> > > > >> On Tue, Mar 25, 2014 at 4:57 PM, Gary Malouf <[email protected] > <javascript:;> > > > > > > >> wrote: > > > >> > For various reasons, our team needs to keep all of our projects on > > the > > > >> same > > > >> > protobuf version. We've now hit a point where we need to upgrade > > > >> protobuf > > > >> > from 2.4.1 to 2.5.0 across the board in our projects and dependent > > > >> > platforms. > > > >> > > > > >> > *Current stack*: Mesos 0.15, Chronos, CDH 4.2.1-MRV1, Spark > > > >> > 0.9-pre-scala-2.10 build off master > > > >> > > > > >> > *Ideal stack after upgrade*: Mesos 0.17, Chronos, CDH5 beta2, > Spark > > > 0.9.1 > > > >> > (hadoop 2.2 build) > > > >> > > > > >> > From what we understand, we need a dependency on Hadoop 2.2 to get > > the > > > >> > necessary protobuf upgrade. From reading Cloudera's documentation > > and > > > >> > multiple google searches, it is not clear to me how we can > construct > > > the > > > >> > stack to continue to work. > > > >> > > > > >> > Has anyone else requested info on getting this combination to > work? > > > From > > > >> > others we've spoken to, they've basically said that we'll be > forced > > to > > > >> use > > > >> > Yarn for Hadoop support in the very near future anyway so we > should > > > >> switch. > > > >> > Since we colocate Spark with out HDFS nodes, it's hard to see how > > we > > > >> would > > > >> > run both Yarn and Mesos on the same servers. > > > >> > > > > > >
