Hi Matei, Thanks for your response. I am using 0.8.1, and yes, It was using protobuf-2.5. Sorry I made a mistake before this email.
I used -Phadoop2-yarn, so I don't find it using pb-2.5, actually, I should use -Pnew-yarn. Thank you Matei. On Mon, Dec 16, 2013 at 4:22 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > Are you using 0.8.1? It will build with protobuf 2.5 instead of 2.4 as > long as you make it depend on Hadoop 2.2. But make sure you build it with > SPARK_HADOOP_VERSION=2.2.0 or whatever. > > Spark 0.8.0 doesn’t support Hadoop 2.2 due to this issue. > > Matei > > On Dec 15, 2013, at 10:25 PM, Azuryy Yu <azury...@gmail.com> wrote: > > > Maybe I am not give a clear description. I am runing Spark on yarn. > instead > > of Mesos. I just want build Spark with protobuf2.5. I am not care about > > Mesos. > > > > I've changed Spark pom.xml to probobuf2.5 manually. > > > > > > > > > > On Mon, Dec 16, 2013 at 2:02 PM, Matei Zaharia <matei.zaha...@gmail.com > >wrote: > > > >> Mesos will almost certainly compile fine with protobuf 2.5. The protobuf > >> compiler and binary format is forward-compatible across releases, it’s > just > >> the Java artifacts that aren’t. You’ll need to ask Mesos to provide a > >> version with protobuf 2.5, and use that with these versions of Hadoop. > >> > >> Matei > >> > >> On Dec 15, 2013, at 7:00 PM, Liu, Raymond <raymond....@intel.com> > wrote: > >> > >>> That issue is for 0.9's solution. > >>> > >>> And if you mean for 0.8.1, when you build against hadoop 2.2 Yarn, > >> protobuf is already using 2.5.0 instead of 2.4.1. so it will works fine > >> with hadoop 2.2 > >>> And regarding on 0.8.1 you build against hadoop 2.2 Yarn, while run > upon > >> mesos... strange combination, I am not sure, might have problem. If have > >> problem, you might need to build mesos against 2.5.0, I don't test > that, if > >> you got time, mind take a test? > >>> > >>> Best Regards, > >>> Raymond Liu > >>> > >>> > >>> -----Original Message----- > >>> From: Liu, Raymond [mailto:raymond....@intel.com] > >>> Sent: Monday, December 16, 2013 10:48 AM > >>> To: dev@spark.incubator.apache.org > >>> Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) > >>> > >>> Hi Azuryy > >>> > >>> Please Check https://spark-project.atlassian.net/browse/SPARK-995 for > >> this protobuf version issue > >>> > >>> Best Regards, > >>> Raymond Liu > >>> > >>> -----Original Message----- > >>> From: Azuryy Yu [mailto:azury...@gmail.com] > >>> Sent: Monday, December 16, 2013 10:30 AM > >>> To: dev@spark.incubator.apache.org > >>> Subject: Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4) > >>> > >>> Hi here, > >>> Do we have plan to upgrade protobuf from 2.4.1 to 2.5.0? PB has some > >> uncompatable API between these two versions. > >>> Hadoop-2.x using protobuf-2.5.0 > >>> > >>> > >>> but if some guys want to run Spark on mesos, then mesos using > >>> protobuf-2.4.1 currently. so we may discuss here for a better solution. > >>> > >>> > >>> > >>> On Mon, Dec 16, 2013 at 7:42 AM, Azuryy Yu <azury...@gmail.com> wrote: > >>> > >>>> Thanks Patrick. > >>>> On 16 Dec 2013 02:43, "Patrick Wendell" <pwend...@gmail.com> wrote: > >>>> > >>>>> You can checkout the docs mentioned in the vote thread. There is also > >>>>> a pre-build binary for hadoop2 that is compiled for YARN 2.2 > >>>>> > >>>>> - Patrick > >>>>> > >>>>> On Sun, Dec 15, 2013 at 4:31 AM, Azuryy Yu <azury...@gmail.com> > wrote: > >>>>>> yarn 2.2, not yarn 0.22, I am so sorry. > >>>>>> > >>>>>> > >>>>>> On Sun, Dec 15, 2013 at 8:31 PM, Azuryy Yu <azury...@gmail.com> > >> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> Spark-0.8.1 supports yarn 0.22 right? where to find the release > note? > >>>>>>> Thanks. > >>>>>>> > >>>>>>> > >>>>>>> On Sun, Dec 15, 2013 at 3:20 AM, Henry Saputra < > >>>>> henry.sapu...@gmail.com>wrote: > >>>>>>> > >>>>>>>> Yeah seems like it. He was ok with our prev release. > >>>>>>>> Let's wait for his reply > >>>>>>>> > >>>>>>>> On Saturday, December 14, 2013, Patrick Wendell wrote: > >>>>>>>> > >>>>>>>>> Henry - from that thread it looks like sebb's concern was > >>>>>>>>> something different than this. > >>>>>>>>> > >>>>>>>>> On Sat, Dec 14, 2013 at 11:08 AM, Henry Saputra < > >>>>>>>> henry.sapu...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>> Hi Patrick, > >>>>>>>>>> > >>>>>>>>>> Yeap I agree, but technically ASF VOTE release on source > >>>>>>>>>> only, > >>>>> there > >>>>>>>>>> even debate about it =), so putting it in the vote staging > >>>>> artifact > >>>>>>>>>> could confuse people because in our case we do package 3rd > >>>>>>>>>> party libraries in the binary jars. > >>>>>>>>>> > >>>>>>>>>> I have sent email to sebb asking clarification about his > >>>>>>>>>> concern > >>>>> in > >>>>>>>>>> general@ list. > >>>>>>>>>> > >>>>>>>>>> - Henry > >>>>>>>>>> > >>>>>>>>>> On Sat, Dec 14, 2013 at 10:56 AM, Patrick Wendell < > >>>>> pwend...@gmail.com > >>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>>> Hey Henry, > >>>>>>>>>>> > >>>>>>>>>>> One thing a lot of people do during the vote is test the > >>>>> binaries and > >>>>>>>>>>> make sure they work. This is really valuable. If you'd like > >>>>>>>>>>> I > >>>>> could > >>>>>>>>>>> add a caveat to the vote thread explaining that we are only > >>>>> voting on > >>>>>>>>>>> the source. > >>>>>>>>>>> > >>>>>>>>>>> - Patrick > >>>>>>>>>>> > >>>>>>>>>>> On Sat, Dec 14, 2013 at 10:40 AM, Henry Saputra < > >>>>>>>>> henry.sapu...@gmail.com> wrote: > >>>>>>>>>>>> Actually we should be fine putting the binaries there as > >>>>>>>>>>>> long > >>>>> as the > >>>>>>>>>>>> VOTE is for the source. > >>>>>>>>>>>> > >>>>>>>>>>>> Let's verify with sebb in the general@ list about his > concern. > >>>>>>>>>>>> > >>>>>>>>>>>> - Henry > >>>>>>>>>>>> > >>>>>>>>>>>> On Sat, Dec 14, 2013 at 10:31 AM, Henry Saputra < > >>>>>>>>> henry.sapu...@gmail.com> wrote: > >>>>>>>>>>>>> Hi Patrick, as sebb has mentioned let's move the binaries > >>>>>>>>>>>>> from > >>>>> the > >>>>>>>>>>>>> voting directory in your people.apache.org directory. > >>>>>>>>>>>>> ASF release voting is for source code and not binaries, > >>>>>>>>>>>>> and technically we provide binaries for convenience. > >>>>>>>>>>>>> > >>>>>>>>>>>>> And add link to the KEYS location in the dist[1] to let > >>>>>>>>>>>>> verify > >>>>>>>>> signatures. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Sorry for the late response to the VOTE thread, guys. > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Henry > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] > >>>>>>>> https://dist.apache.org/repos/dist/release/incubator/spark/KEYS > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Fri, Dec 13, 2013 at 6:37 PM, Patrick Wendell < > >>>>>>>> pwend...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>>> The vote is now closed. This vote passes with 5 PPMC +1's > >>>>>>>>>>>>>> and > >>>>> no 0 > >>>>>>>>> or -1 > >>>>>>>>>>>>>> votes. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> +1 (5 Total) > >>>>>>>>>>>>>> Matei Zaharia* > >>>>>>>>>>>>>> Nick Pentreath* > >>>>>>>>>>>>>> Patrick Wendell* > >>>>>>>>>>>>>> Prashant Sharma* > >>>>>>>>>>>>>> Tom Graves* > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 0 (0 Total) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -1 (0 Total) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * = Binding Vote > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As per the incubator release guide [1] I'll be sending > >>>>>>>>>>>>>> this > >>>>> to the > >>>>>>>>>>>>>> general incubator list for a final vote from IPMC members. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > http://incubator.apache.org/guides/releasemanagement.html#best-practi > >>>>> ce-incubator-release- > >>>>>>>>>>>>>> vote > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Dec 12, 2013 at 8:59 AM, Evan Chan > >>>>>>>>>>>>>> <e...@ooyala.com> > >>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'd be personally fine with a standard workflow of > >>>>> assemble-deps > >>>>>>>> + > >>>>>>>>>>>>>>> packaging just the Spark files as separate packages, if > >>>>>>>>>>>>>>> it > >>>>>>>> speeds up > >>>>>>>>>>>>>>> everyone's development time. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, Dec 11, 2013 at 1:10 PM, Mark Hamstra < > >>>>>>>>> m...@clearstorydata.com > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I don't know how to make sense of the numbers, but > >>>>>>>>>>>>>>>> here's > >>>>> what > >>>>>>>>> I've got > >>>>>>>>>>>>>>>> from a very small sample size. > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>> > >> > >> > >
