Looks like the jar you provided has some missing classes. Try this:
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.3.0",
"org.apache.spark" %% "spark-sql" % "1.3.0" % "provided",
"org.apache.spark" %% "spark-mllib" % "1.3.0" % "provided",
"log4j" % "log4j" % "1.2.15" excludeAll(
ExclusionRule(organization = "com.sun.jdmk"),
ExclusionRule(organization = "com.sun.jmx"),
ExclusionRule(organization = "javax.jms")
)
)
Thanks
Best Regards
On Thu, May 7, 2015 at 11:28 PM, Ganelin, Ilya <[email protected]>
wrote:
> Hi all – I’m attempting to build a project with SBT and run it on Spark
> 1.3 (this previously worked before we upgraded to CDH 5.4 with Spark 1.3).
>
> I have the following in my build.sbt:
>
>
> scalaVersion := "2.10.4"
>
> libraryDependencies ++= Seq(
> "org.apache.spark" %% "spark-core" % "1.3.0" % "provided",
> "org.apache.spark" %% "spark-sql" % "1.3.0" % "provided",
> "org.apache.spark" %% "spark-mllib" % "1.3.0" % "provided",
> "log4j" % "log4j" % "1.2.15" excludeAll(
> ExclusionRule(organization = "com.sun.jdmk"),
> ExclusionRule(organization = "com.sun.jmx"),
> ExclusionRule(organization = "javax.jms")
> )
> )
>
> When I attempt to run this program with sbt run, however, I get the
> following error:
> java.lang.NoClassDefFoundError: org.apache.spark.Partitioner
>
> I don’t explicitly use the Partitioner class anywhere, and this seems to
> indicate some missing Spark libraries on the install. Do I need to confirm
> anything other than the presence of the Spark assembly? I’m on CDH 5.4 and
> I’m able to run the spark-shell without any trouble.
>
> Any help would be much appreciated.
>
> Thank you,
> Ilya Ganelin
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the
> intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of, or
> taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
>
