Starting with Spark 0.9 the protobuf dependency we use is shaded and cannot interfere with other protobuf libaries including those in Hadoop. Not sure what's going on in this case. Would someone who is having this problem post exactly how they are building spark?
- Patrick On Fri, Mar 21, 2014 at 3:49 PM, Aureliano Buendia <buendia...@gmail.com> wrote: > > > > On Tue, Mar 18, 2014 at 12:56 PM, Ognen Duzlevski > <og...@plainvanillagames.com> wrote: >> >> >> On 3/18/14, 4:49 AM, dmpou...@gmail.com wrote: >>> >>> On Sunday, 2 March 2014 19:19:49 UTC+2, Aureliano Buendia wrote: >>>> >>>> Is there a reason for spark using the older akka? >>>> >>>> >>>> >>>> >>>> On Sun, Mar 2, 2014 at 1:53 PM, 1esha <alexey.r...@gmail.com> wrote: >>>> >>>> The problem is in akka remote. It contains files compiled with 2.4.*. >>>> When >>>> >>>> you run it with 2.5.* in classpath it fails like above. >>>> >>>> >>>> >>>> Looks like moving to akka 2.3 will solve this issue. Check this issue - >>>> >>>> >>>> https://www.assembla.com/spaces/akka/tickets/3154-use-protobuf-version-2-5-0#/activity/ticket: >>>> >>>> >>>> Is the solution to exclude the 2.4.*. dependency on protobuf or will >>>> thi produce more complications? >> >> I am not sure I remember what the context was around this but I run 0.9.0 >> with hadoop 2.2.0 just fine. > > > The problem is that spark depends on an older version of akka, which depends > on an older version of protobuf (2.4). > > This means people cannot use protobuf 2.5 with spark. > >> >> Ognen > >
