Just an update on this - since I got the scala notations errors with SBT assembly I went back to the maven build and built with akka 2.3.11 and consumed it on my akka 2.4.1 SBT projects.
Running on the spark cluster this gave problems as the loading order of the dependencies made it so that akka 2.3.11 classes were loaded upfront and at least one class complained about missing methods when being called by the akka 2.4.1 consumer code. In this case was akka.pattern.AskableActorRef. To work around it I had to include into the Spark node's classpath the akka 2.4.0 jars explicitly. These kind of conflicts are the reason why I try to avoid any different version of akka (or any other library) running on the same JVM. For now I've got it running, but's kind of a hack that I'm sure will "hunt me down". I will continue trying to get a running build of Spark with 2.4.0 tnks, Rod On Wednesday, December 2, 2015 at 11:52:07 AM UTC, Rodrigo Boavida wrote: > > Hi Iulian, > > I'm getting build errors using sbt due to the issues you found in the > below thread in July of this year. > > https://mail-archives.apache.org/mod_mbox/spark-dev/201507.mbox/%3CCA+3qhFSJGmZToGmBU1=ivy7kr6eb7k8t6dpz+ibkstihryw...@mail.gmail.com%3E > > Did you get around this meta-notation errors somehow? (also sent you a pvt > msg) > > tnks, > Rod > > On Tuesday, December 1, 2015 at 9:40:17 PM UTC, Iulian Dragoș wrote: >> >> >> >> On Tuesday, December 1, 2015 at 8:52:30 PM UTC+1, Rodrigo Boavida wrote: >>> >>> Hi Iulian, >>> >>> Thanks for the hint. Just to make sure I understood: You actually used >>> sbt assembly to build the spark for server binaries and not just the driver >>> code compilation? Did you also had to change anything in particular in >>> >> terms of merge conflicts or library exclusions for akka 2.4.0 >>> compatibility? >>> >> >> Yes. I remember there were some issues with one or two modules that I had >> to skip, but I don't have the branch at hand. >> >> >>> >>> Tnks. >>> Rod >>> ------------------------------ >>> From: Iulian Dragoș >>> Sent: 01/12/2015 17:28 >>> To: Akka User List >>> Subject: Re: [akka-user] Building Spark with Akka 2.4.0 >>> >>> >>> >>> On Tuesday, December 1, 2015 at 6:09:39 PM UTC+1, drewhk wrote: >>>> >>>> >>>> >>>> On Tue, Dec 1, 2015 at 5:37 PM, Rodrigo Boavida <[email protected]> >>>> wrote: >>>> >>>>> If I revert to Akka 2.3.11 on Scala 2.11 I do not have any problem >>>>> running Spark.Seems something about the way Spark is calling the API that >>>>> internally is creating a conflict? >>>>> >>>>> This happens while creating an actorSystem and some mailbox >>>>> initialization if you look into the log I first posted. >>>>> >>>>> If there are any hints on where to look I will appreciate it. >>>>> >>>> >>>> There is another possible explanation. I vaguely recall our Spark team >>>> having a similar issue and the cause was the maven-shade plugin. It seems >>>> like it modified the contents of the jar ("optimized" it or something) in >>>> a >>>> way that it break it. >>>> >>> >>> Indeed. The problem is with the maven shade plugin, it tries to do too >>> many things when building the assembly, and breaks some classes. >>> >>> I managed to work around it by using `sbt assembly`, which is using >>> another library to build the assembly. >>> >>> iulian >>> >>> >>>> >>>> I'll ask around, maybe someone else can help you here. >>>> >>>> -Endre >>>> >>>> >>>> >>>>> >>>>> Tnks, >>>>> Rod >>>>> >>>>> >>>>> On Tuesday, December 1, 2015 at 3:50:40 PM UTC, Rodrigo Boavida wrote: >>>>> >>>>>> Hi Endre, >>>>>> >>>>>> Spark is supposed to be binary compatible with Scala 2.11. It's >>>>>> actually documented. Here is where they document it, and I'm following >>>>>> the >>>>>> instructions >>>>>> http://spark.apache.org/docs/latest/building-spark.html >>>>>> >>>>>> I actually found that I was missing the -Dscala-2.11 in the maven >>>>>> command when running the dependency, that's why I was getting the >>>>>> previous >>>>>> error. >>>>>> >>>>>> In attach is the result of running the dependencies. >>>>>> >>>>>> tnks, >>>>>> Rod >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, December 1, 2015 at 3:36:51 PM UTC, drewhk wrote: >>>>>> >>>>>> 2.4.0 is not released against 2.10, so that explains it. You probably >>>>>> cannot update to Akka 2.4.0 until Spark is updated to Scala 2.11. >>>>>> >>>>>> -Endre >>>>>> >>>>>> On Tue, Dec 1, 2015 at 4:32 PM, Rodrigo Boavida <[email protected] >>>>>> > wrote: >>>>>> >>>>>> Here is complete dependency resolution. Seems 2.10 dependencies are >>>>>> being pulled from spark-core. >>>>>> >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/com/twitter/chill_2.10/0.5.0/chill_2.10-0.5.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/com/twitter/chill_2.10/0.5.0/chill_2.10-0.5.0.pom >>>>>> >>>>>> (3 KB at 30.5 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.pom >>>>>> >>>>>> (11 KB at 117.4 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project-dist/2.2.0/hadoop-project-dist-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project-dist/2.2.0/hadoop-project-dist-2.2.0.pom >>>>>> >>>>>> (18 KB at 250.3 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.2.0/hadoop-project-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.2.0/hadoo-project-2.2.0.pom >>>>>> >>>>>> (37 KB at 421.9 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-main/2.2.0/hadoop-main-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-main/2.2.0/hadoop-main-2.2.0.pom >>>>>> >>>>>> (17 KB at 224.7 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0.pom >>>>>> >>>>>> (27 KB at 312.4 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/2.2.0/hadoop-annotations-2.2.0.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/2.2.0/hadoop-annotations-2.2.0.pom >>>>>> >>>>>> (3 KB at 39.5 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/commons/commons-math/2.1/commons-math-2.1.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/commons/commons-math/2.1/commons-math-2.1.pom >>>>>> >>>>>> (11 KB at 140.0 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom >>>>>> Downloaded: >>>>>> https://repo1.maven.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom >>>>>> >>>>>> (31 KB at 369.3 KB/sec) >>>>>> Downloading: >>>>>> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.2.0/hadoop-auth-2.2.0.pom >>>>>> Downloaded: >>>>>> >>>>>> >>> [The entire original message is not included.] >>> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
