Re: 1.3 Build Error with Scala-2.11
did you run dev/change-version-to-2.11.sh before compiling? When I ran this on current master, it mostly worked: dev/change-version-to-2.11.sh mvn -Pyarn -Phadoop-2.4 -Pscala-2.11 -DskipTests clean package There was a failure in building catalyst, but core built just fine for me. The error I got was: [INFO] [INFO] Building Spark Project Catalyst 1.4.0-SNAPSHOT [INFO] [WARNING] The POM for org.scalamacros:quasiquotes_2.11:jar:2.0.1 is missing, no dependency information available I'm not sure if catalyst is supposed to work w/ scala-2.11 or not ... I wouldn't be surprised if the way macros should be used has changed, but its not listed explicitly in the docs as being incompatible: http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211 On Tue, Apr 7, 2015 at 12:00 AM, mjhb sp...@mjhb.com wrote: I even deleted my local maven repository (.m2) but still stuck when attempting to build w/ Scala-2.11: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11449.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
Yes - ran dev/change-version-to-2.11.sh But was missing -Dscala-2.11 on mvn command after a -2.10 build. Building successfully again now after adding that. On Tue, Apr 7, 2015 at 7:04 PM Imran Rashid iras...@cloudera.com wrote: did you run dev/change-version-to-2.11.sh before compiling? When I ran this on current master, it mostly worked: dev/change-version-to-2.11.sh mvn -Pyarn -Phadoop-2.4 -Pscala-2.11 -DskipTests clean package There was a failure in building catalyst, but core built just fine for me. The error I got was: [INFO] [INFO] Building Spark Project Catalyst 1.4.0-SNAPSHOT [INFO] [WARNING] The POM for org.scalamacros:quasiquotes_2.11:jar:2.0.1 is missing, no dependency information available I'm not sure if catalyst is supposed to work w/ scala-2.11 or not ... I wouldn't be surprised if the way macros should be used has changed, but its not listed explicitly in the docs as being incompatible: http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211 On Tue, Apr 7, 2015 at 12:00 AM, mjhb sp...@mjhb.com wrote: I even deleted my local maven repository (.m2) but still stuck when attempting to build w/ Scala-2.11: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11449.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
Similar problem on 1.2 branch: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11442.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
I'm killing zinc (if it's running) before running each build attempt. Trying to build as clean as possible. On Mon, Apr 6, 2015 at 7:31 PM Patrick Wendell pwend...@gmail.com wrote: What if you don't run zinc? I.e. just download maven and run that mvn package It might take longer, but I wonder if it will work. On Mon, Apr 6, 2015 at 10:26 PM, mjhb sp...@mjhb.com wrote: Similar problem on 1.2 branch: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -- View this message in context: http://apache-spark- developers-list.1001551.n3.nabble.com/1-3-Build-Error- with-Scala-2-11-tp11441p11442.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
The issue is that if you invoke build/mvn it will start zinc again if it sees that it is killed. The absolute most sterile thing to do is this: 1. Kill any zinc processes. 2. Clean up spark git clean -fdx (WARNING: this will delete any staged changes you have, if you have code modifications or extra files around) 3. Run the 2.11 script to change the versions. 4. Run mvn package with maven that you installed on your machine. On Mon, Apr 6, 2015 at 10:43 PM, Marty Bower sp...@mjhb.com wrote: I'm killing zinc (if it's running) before running each build attempt. Trying to build as clean as possible. On Mon, Apr 6, 2015 at 7:31 PM Patrick Wendell pwend...@gmail.com wrote: What if you don't run zinc? I.e. just download maven and run that mvn package It might take longer, but I wonder if it will work. On Mon, Apr 6, 2015 at 10:26 PM, mjhb sp...@mjhb.com wrote: Similar problem on 1.2 branch: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11442.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
One thing that I think can cause issues is if you run build/mvn with Scala 2.10, then try to run it with 2.11, since I think we may store some downloaded jars relating to zinc that will get screwed up. Not sure that's what is happening, just an idea. On Mon, Apr 6, 2015 at 10:54 PM, Patrick Wendell pwend...@gmail.com wrote: The issue is that if you invoke build/mvn it will start zinc again if it sees that it is killed. The absolute most sterile thing to do is this: 1. Kill any zinc processes. 2. Clean up spark git clean -fdx (WARNING: this will delete any staged changes you have, if you have code modifications or extra files around) 3. Run the 2.11 script to change the versions. 4. Run mvn package with maven that you installed on your machine. On Mon, Apr 6, 2015 at 10:43 PM, Marty Bower sp...@mjhb.com wrote: I'm killing zinc (if it's running) before running each build attempt. Trying to build as clean as possible. On Mon, Apr 6, 2015 at 7:31 PM Patrick Wendell pwend...@gmail.com wrote: What if you don't run zinc? I.e. just download maven and run that mvn package It might take longer, but I wonder if it will work. On Mon, Apr 6, 2015 at 10:26 PM, mjhb sp...@mjhb.com wrote: Similar problem on 1.2 branch: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11442.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
I resorted to deleting the spark directory between each build earlier today (attempting maximum sterility) and then re-cloning from github and switching to the 1.2 or 1.3 branch. Does anything persist outside of the spark directory? Are you able to build either 1.2 or 1.3 w/ Scala-2.11? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11447.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
What if you don't run zinc? I.e. just download maven and run that mvn package It might take longer, but I wonder if it will work. On Mon, Apr 6, 2015 at 10:26 PM, mjhb sp...@mjhb.com wrote: Similar problem on 1.2 branch: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.2.3-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.2.3-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.10:jar:1.2.3-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11442.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
The only think that can persist outside of Spark is if there is still a live Zinc process. We took care to make sure this was a generally stateless mechanism. Both the 1.2.X and 1.3.X releases are built with Scala 2.11 for packaging purposes. And these have been built as recently as in the last few days, since we are voting on 1.2.2 and 1.3.1. However there could be issues that only affect certain environments. - Patrick On Mon, Apr 6, 2015 at 11:02 PM, mjhb sp...@mjhb.com wrote: I resorted to deleting the spark directory between each build earlier today (attempting maximum sterility) and then re-cloning from github and switching to the 1.2 or 1.3 branch. Does anything persist outside of the spark directory? Are you able to build either 1.2 or 1.3 w/ Scala-2.11? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11447.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: 1.3 Build Error with Scala-2.11
Hmm.. Make sure you are building with the right flags. I think you need to pass -Dscala-2.11 to maven. Take a look at the upstream docs - on my phone now so can't easily access. On Apr 7, 2015 1:01 AM, mjhb sp...@mjhb.com wrote: I even deleted my local maven repository (.m2) but still stuck when attempting to build w/ Scala-2.11: [ERROR] Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project spark-core_2.11: Could not resolve dependencies for project org.apache.spark:spark-core_2.11:jar:1.3.2-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.10:jar:1.3.2-SNAPSHOT: Could not find artifact org.apache.spark:spark-network-common_2.10:jar:1.3.2-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/1-3-Build-Error-with-Scala-2-11-tp11441p11449.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
