Hello Koert, That did not work. I specified it in my email already. But I figured a way around it by excluding akka dependencies
Shivani On Tue, Apr 29, 2014 at 12:37 PM, Koert Kuipers <ko...@tresata.com> wrote: > you need to merge reference.conf files and its no longer an issue. > > see the Build for for spark itself: > case "reference.conf" => MergeStrategy.concat > > > On Tue, Apr 29, 2014 at 3:32 PM, Shivani Rao <raoshiv...@gmail.com> wrote: > >> Hello folks, >> >> I was going to post this question to spark user group as well. If you >> have any leads on how to solve this issue please let me know: >> >> I am trying to build a basic spark project (spark depends on akka) and I >> am trying to create a fatjar using sbt assembly. The goal is to run the >> fatjar via commandline as follows: >> java -cp "path to my spark fatjar" mainclassname >> >> I encountered deduplication errors in the following akka libraries during >> sbt assembly >> akka-remote_2.10-2.2.3.jar with akka-remote_2.10-2.2.3-shaded-protobuf.jar >> akka-actor_2.10-2.2.3.jar with akka-actor_2.10-2.2.3-shaded-protobuf.jar >> >> I resolved them by using MergeStrategy.first and that helped with a >> successful compilation of the sbt assembly command. But for some or the >> other configuration parameter in the akka kept throwing up with the >> following message >> >> "Exception in thread "main" com.typesafe.config.ConfigException$Missing: >> No configuration setting found for key" >> >> I then used MergeStrategy.concat for "reference.conf" and I started >> getting this repeated error >> >> Exception in thread "main" com.typesafe.config.ConfigException$Missing: >> No configuration setting found for key 'akka.version'. >> >> I noticed that akka.version is only in the akka-actor jars and not in the >> akka-remote. The resulting reference.conf (in my final fat jar) does not >> contain akka.version either. So the strategy is not working. >> >> There are several things I could try >> >> a) Use the following dependency https://github.com/sbt/sbt-proguard >> b) Write a build.scala to handle merging of reference.conf >> >> https://spark-project.atlassian.net/browse/SPARK-395 >> >> http://letitcrash.com/post/21025950392/howto-sbt-assembly-vs-reference-conf >> >> c) Create a reference.conf by merging all akka configurations and then >> passing it in my java -cp command as shown below >> >> java -cp <jar-name> -DConfig.file=<config> >> >> The main issue is that if I run the spark jar as "sbt run" there are no >> errors in accessing any of the akka configuration parameters. It is only >> when I run it via command line (java -cp <jar-name> classname) that I >> encounter the error. >> >> Which of these is a long term fix to akka issues? For now, I removed the >> akka dependencies and that solved the problem, but I know that is not a >> long term solution >> >> Regards, >> Shivani >> >> -- >> Software Engineer >> Analytics Engineering Team@ Box >> Mountain View, CA >> > > -- Software Engineer Analytics Engineering Team@ Box Mountain View, CA
