Re: Upgrade the scala code using the most updated Spark version

Hi, Thanks everybody to help me to solve my problem :) As Zhu said, I had to use mapPartitionsWithIndex in my code. Thanks, Have a nice day, Anahita On Wed, Mar 29, 2017 at 2:51 AM, Shixiong(Ryan) Zhu wrote: > mapPartitionsWithSplit was removed in Spark 2.0.0. You

Re: Upgrade the scala code using the most updated Spark version

mapPartitionsWithSplit was removed in Spark 2.0.0. You can use mapPartitionsWithIndex instead. On Tue, Mar 28, 2017 at 3:52 PM, Anahita Talebi wrote: > Thanks. > I tried this one, as well. Unfortunately I still get the same error. > > > On Wednesday, March 29, 2017,

Re: Upgrade the scala code using the most updated Spark version

Thanks. I tried this one, as well. Unfortunately I still get the same error. On Wednesday, March 29, 2017, Marco Mistroni wrote: > 1.7.5 > > On 28 Mar 2017 10:10 pm, "Anahita Talebi" >

Re: Upgrade the scala code using the most updated Spark version

1.7.5 On 28 Mar 2017 10:10 pm, "Anahita Talebi" wrote: > Hi, > > Thanks for your answer. > What is the version of "org.slf4j" % "slf4j-api" in your sbt file? > I think the problem might come from this part. > > On Tue, Mar 28, 2017 at 11:02 PM, Marco Mistroni

Re: Upgrade the scala code using the most updated Spark version

Hello again, I just tried to change the version to 3.0.0 and remove the libraries breeze, netlib and scoopt but I still get the same error. On Tue, Mar 28, 2017 at 11:02 PM, Marco Mistroni wrote: > Hello > uhm ihave a project whose build,sbt is closest to yours, where i am

Re: Upgrade the scala code using the most updated Spark version

Hi, Thanks for your answer. What is the version of "org.slf4j" % "slf4j-api" in your sbt file? I think the problem might come from this part. On Tue, Mar 28, 2017 at 11:02 PM, Marco Mistroni wrote: > Hello > uhm ihave a project whose build,sbt is closest to yours, where i

Re: Upgrade the scala code using the most updated Spark version

Hello uhm ihave a project whose build,sbt is closest to yours, where i am using spark 2.1, scala 2.11 and scalatest (i upgraded to 3.0.0) and it works fine in my projects though i don thave any of the following libraries that you mention - breeze - netlib,all - scoopt hth On Tue, Mar 28, 2017

Re: Upgrade the scala code using the most updated Spark version

Hi, Thanks for your answer. I first changed the scala version to 2.11.8 and kept the spark version 1.5.2 (old version). Then I changed the scalatest version into "3.0.1". With this configuration, I could run the code and compile it and generate the .jar file. When I changed the spark version

Re: Upgrade the scala code using the most updated Spark version

I personally never add the _scala version to the dependency but always crosscompile. This seems to be cleanest. Additionally Spark dependencies and hadoop dependencies should be provided not compile. Scalatest seems to be outdated. I would also not use a local repo, but either an artefact

Re: Upgrade the scala code using the most updated Spark version

Hi, Thanks for your answer. I just changes the sbt file and set the scala version to 2.10.4 But I still get the same error [info] Compiling 4 Scala sources to /Users/atalebi/Desktop/new_version_proxcocoa-master/target/scala-2.10/classes... [error]

Re: Upgrade the scala code using the most updated Spark version

Hello that looks to me like there's something dodgy withyour Scala installation Though Spark 2.0 is built on Scala 2.11, it still support 2.10... i suggest you change one thing at a time in your sbt First Spark version. run it and see if it works Then amend the scala version hth marco On Tue,

Re: Upgrade the scala code using the most updated Spark version

Hello, Thanks you all for your informative answers. I actually changed the scala version to the 2.11.8 and spark version into 2.1.0 in the build.sbt Except for these two guys (scala and spark version), I kept the same values for the rest in the build.sbt file.

Re: Upgrade the scala code using the most updated Spark version

Adding to advices given by others ... Spark 2.1.0 works with Scala 2.11, so set: scalaVersion := "2.11.8" When you see something like: "org.apache.spark" % "spark-core_2.10" % "1.5.2" that means that library `spark-core` is compiled against Scala 2.10, so you would have to change that to

Re: Upgrade the scala code using the most updated Spark version

it yo On Jörn Franke , Mar 28, 2017 12:11 AM wrote:Usually you define the dependencies to the Spark library as provided. You also seem to mix different Spark versions which should be avoided.The Hadoop library seems to be outdated and should also only be provided.The other

Re: Upgrade the scala code using the most updated Spark version

check these versions function create_build_sbt_file { BUILD_SBT_FILE=${GEN_APPSDIR}/scala/${APPLICATION}/build.sbt [ -f ${BUILD_SBT_FILE} ] && rm -f ${BUILD_SBT_FILE} cat >> $BUILD_SBT_FILE << ! lazy val root = (project in file(".")). settings( name :=

Re: Upgrade the scala code using the most updated Spark version

Usually you define the dependencies to the Spark library as provided. You also seem to mix different Spark versions which should be avoided. The Hadoop library seems to be outdated and should also only be provided. The other dependencies you could assemble in a fat jar. > On 27 Mar 2017, at