Are you using 0.8.1? It will build with protobuf 2.5 instead of 2.4 as long as you make it depend on Hadoop 2.2. But make sure you build it with SPARK_HADOOP_VERSION=2.2.0 or whatever.
Spark 0.8.0 doesn’t support Hadoop 2.2 due to this issue. Matei On Dec 15, 2013, at 10:25 PM, Azuryy Yu <azury...@gmail.com> wrote: > Maybe I am not give a clear description. I am runing Spark on yarn. instead > of Mesos. I just want build Spark with protobuf2.5. I am not care about > Mesos. > > I've changed Spark pom.xml to probobuf2.5 manually. > > > > > On Mon, Dec 16, 2013 at 2:02 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > >> Mesos will almost certainly compile fine with protobuf 2.5. The protobuf >> compiler and binary format is forward-compatible across releases, it’s just >> the Java artifacts that aren’t. You’ll need to ask Mesos to provide a >> version with protobuf 2.5, and use that with these versions of Hadoop. >> >> Matei >> >> On Dec 15, 2013, at 7:00 PM, Liu, Raymond <raymond....@intel.com> wrote: >> >>> That issue is for 0.9's solution. >>> >>> And if you mean for 0.8.1, when you build against hadoop 2.2 Yarn, >> protobuf is already using 2.5.0 instead of 2.4.1. so it will works fine >> with hadoop 2.2 >>> And regarding on 0.8.1 you build against hadoop 2.2 Yarn, while run upon >> mesos... strange combination, I am not sure, might have problem. If have >> problem, you might need to build mesos against 2.5.0, I don't test that, if >> you got time, mind take a test? >>> >>> Best Regards, >>> Raymond Liu >>> >>> >>> -----Original Message----- >>> From: Liu, Raymond [mailto:raymond....@intel.com] >>> Sent: Monday, December 16, 2013 10:48 AM >>> To: dev@spark.incubator.apache.org >>> Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) >>> >>> Hi Azuryy >>> >>> Please Check https://spark-project.atlassian.net/browse/SPARK-995 for >> this protobuf version issue >>> >>> Best Regards, >>> Raymond Liu >>> >>> -----Original Message----- >>> From: Azuryy Yu [mailto:azury...@gmail.com] >>> Sent: Monday, December 16, 2013 10:30 AM >>> To: dev@spark.incubator.apache.org >>> Subject: Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) >>> >>> Hi here, >>> Do we have plan to upgrade protobuf from 2.4.1 to 2.5.0? PB has some >> uncompatable API between these two versions. >>> Hadoop-2.x using protobuf-2.5.0 >>> >>> >>> but if some guys want to run Spark on mesos, then mesos using >>> protobuf-2.4.1 currently. so we may discuss here for a better solution. >>> >>> >>> >>> On Mon, Dec 16, 2013 at 7:42 AM, Azuryy Yu <azury...@gmail.com> wrote: >>> >>>> Thanks Patrick. >>>> On 16 Dec 2013 02:43, "Patrick Wendell" <pwend...@gmail.com> wrote: >>>> >>>>> You can checkout the docs mentioned in the vote thread. There is also >>>>> a pre-build binary for hadoop2 that is compiled for YARN 2.2 >>>>> >>>>> - Patrick >>>>> >>>>> On Sun, Dec 15, 2013 at 4:31 AM, Azuryy Yu <azury...@gmail.com> wrote: >>>>>> yarn 2.2, not yarn 0.22, I am so sorry. >>>>>> >>>>>> >>>>>> On Sun, Dec 15, 2013 at 8:31 PM, Azuryy Yu <azury...@gmail.com> >> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Spark-0.8.1 supports yarn 0.22 right? where to find the release note? >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> On Sun, Dec 15, 2013 at 3:20 AM, Henry Saputra < >>>>> henry.sapu...@gmail.com>wrote: >>>>>>> >>>>>>>> Yeah seems like it. He was ok with our prev release. >>>>>>>> Let's wait for his reply >>>>>>>> >>>>>>>> On Saturday, December 14, 2013, Patrick Wendell wrote: >>>>>>>> >>>>>>>>> Henry - from that thread it looks like sebb's concern was >>>>>>>>> something different than this. >>>>>>>>> >>>>>>>>> On Sat, Dec 14, 2013 at 11:08 AM, Henry Saputra < >>>>>>>> henry.sapu...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>> Hi Patrick, >>>>>>>>>> >>>>>>>>>> Yeap I agree, but technically ASF VOTE release on source >>>>>>>>>> only, >>>>> there >>>>>>>>>> even debate about it =), so putting it in the vote staging >>>>> artifact >>>>>>>>>> could confuse people because in our case we do package 3rd >>>>>>>>>> party libraries in the binary jars. >>>>>>>>>> >>>>>>>>>> I have sent email to sebb asking clarification about his >>>>>>>>>> concern >>>>> in >>>>>>>>>> general@ list. >>>>>>>>>> >>>>>>>>>> - Henry >>>>>>>>>> >>>>>>>>>> On Sat, Dec 14, 2013 at 10:56 AM, Patrick Wendell < >>>>> pwend...@gmail.com >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>>> Hey Henry, >>>>>>>>>>> >>>>>>>>>>> One thing a lot of people do during the vote is test the >>>>> binaries and >>>>>>>>>>> make sure they work. This is really valuable. If you'd like >>>>>>>>>>> I >>>>> could >>>>>>>>>>> add a caveat to the vote thread explaining that we are only >>>>> voting on >>>>>>>>>>> the source. >>>>>>>>>>> >>>>>>>>>>> - Patrick >>>>>>>>>>> >>>>>>>>>>> On Sat, Dec 14, 2013 at 10:40 AM, Henry Saputra < >>>>>>>>> henry.sapu...@gmail.com> wrote: >>>>>>>>>>>> Actually we should be fine putting the binaries there as >>>>>>>>>>>> long >>>>> as the >>>>>>>>>>>> VOTE is for the source. >>>>>>>>>>>> >>>>>>>>>>>> Let's verify with sebb in the general@ list about his concern. >>>>>>>>>>>> >>>>>>>>>>>> - Henry >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Dec 14, 2013 at 10:31 AM, Henry Saputra < >>>>>>>>> henry.sapu...@gmail.com> wrote: >>>>>>>>>>>>> Hi Patrick, as sebb has mentioned let's move the binaries >>>>>>>>>>>>> from >>>>> the >>>>>>>>>>>>> voting directory in your people.apache.org directory. >>>>>>>>>>>>> ASF release voting is for source code and not binaries, >>>>>>>>>>>>> and technically we provide binaries for convenience. >>>>>>>>>>>>> >>>>>>>>>>>>> And add link to the KEYS location in the dist[1] to let >>>>>>>>>>>>> verify >>>>>>>>> signatures. >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for the late response to the VOTE thread, guys. >>>>>>>>>>>>> >>>>>>>>>>>>> - Henry >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>> https://dist.apache.org/repos/dist/release/incubator/spark/KEYS >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Dec 13, 2013 at 6:37 PM, Patrick Wendell < >>>>>>>> pwend...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>>>> The vote is now closed. This vote passes with 5 PPMC +1's >>>>>>>>>>>>>> and >>>>> no 0 >>>>>>>>> or -1 >>>>>>>>>>>>>> votes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> +1 (5 Total) >>>>>>>>>>>>>> Matei Zaharia* >>>>>>>>>>>>>> Nick Pentreath* >>>>>>>>>>>>>> Patrick Wendell* >>>>>>>>>>>>>> Prashant Sharma* >>>>>>>>>>>>>> Tom Graves* >>>>>>>>>>>>>> >>>>>>>>>>>>>> 0 (0 Total) >>>>>>>>>>>>>> >>>>>>>>>>>>>> -1 (0 Total) >>>>>>>>>>>>>> >>>>>>>>>>>>>> * = Binding Vote >>>>>>>>>>>>>> >>>>>>>>>>>>>> As per the incubator release guide [1] I'll be sending >>>>>>>>>>>>>> this >>>>> to the >>>>>>>>>>>>>> general incubator list for a final vote from IPMC members. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> http://incubator.apache.org/guides/releasemanagement.html#best-practi >>>>> ce-incubator-release- >>>>>>>>>>>>>> vote >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Dec 12, 2013 at 8:59 AM, Evan Chan >>>>>>>>>>>>>> <e...@ooyala.com> >>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'd be personally fine with a standard workflow of >>>>> assemble-deps >>>>>>>> + >>>>>>>>>>>>>>> packaging just the Spark files as separate packages, if >>>>>>>>>>>>>>> it >>>>>>>> speeds up >>>>>>>>>>>>>>> everyone's development time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Dec 11, 2013 at 1:10 PM, Mark Hamstra < >>>>>>>>> m...@clearstorydata.com >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I don't know how to make sense of the numbers, but >>>>>>>>>>>>>>>> here's >>>>> what >>>>>>>>> I've got >>>>>>>>>>>>>>>> from a very small sample size. >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>> >> >>
