Thanks Pankaj for the assembly plugin tip.
Yes there is a version mismatch of akka actor between Spark 1.1.1 and
akka-http/akka-stream (2.2.3 versus 2.3.x).
After some digging, I see 4 options for this problem (in case others
encounter it):
1) Upgrade to Spark 1.2.0, the same code will work (not possible for me
since I want to use Datastax connector)
2) Make a custom build of Spark 1.1.1
3) Use play instead with akka actor 2.2.3 (play 2.2.3 for instance)
4) Wait for Datastax connector 1.2.0 (released on 31th January 2015)
I currently trying option 3
Thank you all for your help
On Sat, Jan 3, 2015 at 4:11 AM, Pankaj Narang [via Apache Spark User List] <
ml-node+s1001560n20950...@n3.nabble.com> wrote:
> Like before I get a java.lang.NoClassDefFoundError:
> akka/stream/FlowMaterializer$
>
> This can be solved using assembly plugin. you need to enable assembly
> plugin in global plugins
>
> C:\Users\infoshore\.sbt\0.13\plugins
> add a line in plugins.sbt addSbtPlugin("com.eed3si9n" % "sbt-assembly" %
> "0.11.0")
>
>
>
> and then add the following lines in build.sbt
>
> import AssemblyKeys._ // put this at the top of the file
>
> seq(assemblySettings: _*)
>
> Also in the bottom dont forget to add
>
> assemblySettings
>
> mergeStrategy in assembly := {
> case m if m.toLowerCase.endsWith("manifest.mf") =>
> MergeStrategy.discard
> case m if m.toLowerCase.matches("meta-inf.*\\.sf$") =>
> MergeStrategy.discard
> case "log4j.properties" =>
> MergeStrategy.discard
> case m if m.toLowerCase.startsWith("meta-inf/services/") =>
> MergeStrategy.filterDistinctLines
> case "reference.conf" =>
> MergeStrategy.concat
> case _ =>
> MergeStrategy.first
> }
>
>
> Now in your sbt run sbt assembly that will create the jar which can be run
> without --jars options
> as this will be a uber jar containing all jars
>
>
>
> Also nosuchmethod exception is thrown when there is difference in versions
> of complied and runtime versions.
>
> What is the version of spark you are using ? You need to use same version
> in build.sbt
>
>
> Here is your build.sbt
>
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
> //exclude("com.typesafe", "config")
>
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>
> libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core"
> % "2.1.3"
>
> libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector"
> % "1.1.0" withSources() withJavadoc()
>
> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
> "2.0.5"
>
> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>
>
> and your error is Exception in thread "main" java.lang.NoSuchMethodError:
> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>
> at
> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>
> I think there is version mismatch on the jars you use at runtime
>
>
> If you need more help add me on skype pankaj.narang
>
>
> ---Pankaj
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20950.html
> To unsubscribe from NoSuchMethodError:
> com.typesafe.config.Config.getDuration with akka-http/akka-stream, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=20926&code=Y2hyaXN0b3BoZS5iaWxsaWFyZEBnbWFpbC5jb218MjA5MjZ8LTE2ODA2NTAwMDk=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20988.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
