[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-195119739
  
I believe that the earlier PySpark test failures were related to 
https://issues.apache.org/jira/browse/SPARK-6047. I've opened 
https://github.com/bartdag/py4j/pull/196 to try to fix this upstream. While we 
wait for that chain of fixes to go in, though, I'd like to get the bulk of the 
MiMa changes here wrapped up and merged, so I'm going to re-scope this ticket 
to only remove the use of assemblies from MiMa and leave it in for the rest of 
the tess.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-195088675
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52857/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-195088632
  
**[Test build #52857 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52857/consoleFull)**
 for PR 11178 at commit 
[`4756a1e`](https://github.com/apache/spark/commit/4756a1e582075ba8364da4e325b77c9f6b998602).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-195088673
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-195077598
  
**[Test build #52857 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52857/consoleFull)**
 for PR 11178 at commit 
[`4756a1e`](https://github.com/apache/spark/commit/4756a1e582075ba8364da4e325b77c9f6b998602).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55755114
  
--- Diff: dev/mima ---
@@ -24,24 +24,63 @@ set -e
 FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
 cd "$FWDIR"
 
-echo -e "q\n" | build/sbt oldDeps/update
-rm -f .generated-mima*
+# All default projects except for assemblies:
+SPARK_PROJECTS=(
+  "common/network-common"
+  "common/network-shuffle"
+  "common/network-yarn"
+  "common/sketch"
+  "common/tags"
+  "common/unsafe"
+  "core"
+  "docker-integration-tests"
+  "examples"
+  "external/akka"
+  "external/flume"
+  "external/flume-sink"
+  "external/kafka"
+  "external/mqtt"
+  "external/twitter"
+  "external/zeromq"
+  "extras/kinesis-asl"
--- End diff --

It turns out that we don't support MiMa for the Maven build anyways, so I 
can just safely pull this in from one of SBT's `fullClasspath` outputs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55754957
  
--- Diff: project/SparkBuild.scala ---
@@ -384,18 +384,19 @@ object OldDeps {
 
   lazy val project = Project("oldDeps", file("dev"), settings = 
oldDepsSettings)
 
-  def versionArtifact(id: String): Option[sbt.ModuleID] = {
-val fullId = id + "_2.11"
-Some("org.apache.spark" % fullId % "1.2.0")
-  }
-
   def oldDepsSettings() = Defaults.coreDefaultSettings ++ Seq(
 name := "old-deps",
 scalaVersion := "2.10.5",
-libraryDependencies := Seq("spark-streaming-mqtt", 
"spark-streaming-zeromq",
-  "spark-streaming-flume", "spark-streaming-twitter",
-  "spark-streaming", "spark-mllib", "spark-graphx",
-  "spark-core").map(versionArtifact(_).get intransitive())
+libraryDependencies := Seq(
+  "spark-streaming-mqtt",
+  "spark-streaming-zeromq",
+  "spark-streaming-flume",
+  "spark-streaming-twitter",
+  "spark-streaming",
+  "spark-mllib",
+  "spark-graphx",
+  "spark-core"
+).map(id => "org.apache.spark" % (id + "_2.11") % "1.2.0")
--- End diff --

I'm going to post a larger followup in a few minutes with diffs to explain 
this fully, but in a nutshell the GenerateMiMaIgnore was hitting many spurious 
errors because it couldn't load / reflect on classes since their dependencies 
were missing; pulling in these transitive deps fixes that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55652534
  
--- Diff: project/SparkBuild.scala ---
@@ -384,18 +384,19 @@ object OldDeps {
 
   lazy val project = Project("oldDeps", file("dev"), settings = 
oldDepsSettings)
 
-  def versionArtifact(id: String): Option[sbt.ModuleID] = {
-val fullId = id + "_2.11"
-Some("org.apache.spark" % fullId % "1.2.0")
-  }
-
   def oldDepsSettings() = Defaults.coreDefaultSettings ++ Seq(
 name := "old-deps",
 scalaVersion := "2.10.5",
-libraryDependencies := Seq("spark-streaming-mqtt", 
"spark-streaming-zeromq",
-  "spark-streaming-flume", "spark-streaming-twitter",
-  "spark-streaming", "spark-mllib", "spark-graphx",
-  "spark-core").map(versionArtifact(_).get intransitive())
+libraryDependencies := Seq(
+  "spark-streaming-mqtt",
+  "spark-streaming-zeromq",
+  "spark-streaming-flume",
+  "spark-streaming-twitter",
+  "spark-streaming",
+  "spark-mllib",
+  "spark-graphx",
+  "spark-core"
+).map(id => "org.apache.spark" % (id + "_2.11") % "1.2.0")
--- End diff --

intransitive at the end is no longer needed ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55642612
  
--- Diff: 
launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -146,10 +146,38 @@ void addOptionString(List cmd, String 
options) {
 boolean isTesting = "1".equals(getenv("SPARK_TESTING"));
 if (prependClasses || isTesting) {
   String scala = getScalaVersion();
-  List projects = Arrays.asList("core", "repl", "mllib", 
"graphx",
-"streaming", "tools", "sql/catalyst", "sql/core", "sql/hive", 
"sql/hive-thriftserver",
-"yarn", "launcher",
-"common/network-common", "common/network-shuffle", 
"common/network-yarn");
+  // All projects except assemblies:
+  List projects = Arrays.asList(
+"common/network-common",
+"common/network-shuffle",
+"common/network-yarn",
+"common/sketch",
+"common/tags",
+"common/unsafe",
+"core",
+"docker-integration-tests",
+"examples",
+"external/akka",
+"external/flume",
+"external/flume-sink",
+"external/kafka",
+"external/mqtt",
+"external/twitter",
+"external/zeromq",
+"extras/kinesis-asl",
--- End diff --

Same here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r55642583
  
--- Diff: dev/mima ---
@@ -24,24 +24,63 @@ set -e
 FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
 cd "$FWDIR"
 
-echo -e "q\n" | build/sbt oldDeps/update
-rm -f .generated-mima*
+# All default projects except for assemblies:
+SPARK_PROJECTS=(
+  "common/network-common"
+  "common/network-shuffle"
+  "common/network-yarn"
+  "common/sketch"
+  "common/tags"
+  "common/unsafe"
+  "core"
+  "docker-integration-tests"
+  "examples"
+  "external/akka"
+  "external/flume"
+  "external/flume-sink"
+  "external/kafka"
+  "external/mqtt"
+  "external/twitter"
+  "external/zeromq"
+  "extras/kinesis-asl"
--- End diff --

BTW extras/* is now in external/, and docker* directories are also in 
external/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191098515
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52293/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191098514
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191098324
  
**[Test build #52293 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52293/consoleFull)**
 for PR 11178 at commit 
[`64330d6`](https://github.com/apache/spark/commit/64330d622331e56939dbabb0583523b5a56ef4bf).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191061789
  
**[Test build #52293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52293/consoleFull)**
 for PR 11178 at commit 
[`64330d6`](https://github.com/apache/spark/commit/64330d622331e56939dbabb0583523b5a56ef4bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191060308
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191058602
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191058605
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52284/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191058512
  
**[Test build #52284 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52284/consoleFull)**
 for PR 11178 at commit 
[`64330d6`](https://github.com/apache/spark/commit/64330d622331e56939dbabb0583523b5a56ef4bf).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191032108
  
Didn't see anything that caught my eye, so if tests pass this LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-191025386
  
**[Test build #52284 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52284/consoleFull)**
 for PR 11178 at commit 
[`64330d6`](https://github.com/apache/spark/commit/64330d622331e56939dbabb0583523b5a56ef4bf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190984002
  
**[Test build #52265 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52265/consoleFull)**
 for PR 11178 at commit 
[`a73118e`](https://github.com/apache/spark/commit/a73118e49178cb0a282058d14383f219d028265f).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190984081
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52265/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190984078
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190979665
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52262/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190979527
  
**[Test build #52262 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52262/consoleFull)**
 for PR 11178 at commit 
[`902b1b7`](https://github.com/apache/spark/commit/902b1b79f8ccfc2eaf1d4b25be64e947250665b1).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190979655
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190975675
  
**[Test build #52265 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52265/consoleFull)**
 for PR 11178 at commit 
[`a73118e`](https://github.com/apache/spark/commit/a73118e49178cb0a282058d14383f219d028265f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-190971242
  
**[Test build #52262 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52262/consoleFull)**
 for PR 11178 at commit 
[`902b1b7`](https://github.com/apache/spark/commit/902b1b79f8ccfc2eaf1d4b25be64e947250665b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185476411
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185476392
  
**[Test build #51449 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51449/consoleFull)**
 for PR 11178 at commit 
[`906d8c8`](https://github.com/apache/spark/commit/906d8c80c15062eac4f2cd58e055a7789089).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185476413
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51449/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185471138
  
**[Test build #51449 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51449/consoleFull)**
 for PR 11178 at commit 
[`906d8c8`](https://github.com/apache/spark/commit/906d8c80c15062eac4f2cd58e055a7789089).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185463493
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185451762
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185451763
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51448/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185366797
  
```
Failed to find Spark assembly in 
/home/jenkins/workspace/SparkPullRequestBuilder/assembly/target/scala-2.11.
You need to build Spark before running this program.
```

I guess that the PySpark test scripts also need to set 
`SPARK_PREPEND_CLASSES` and a couple of other environment variables. I'll see 
about fixing that now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185360769
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185360779
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51435/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185360335
  
**[Test build #51435 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51435/consoleFull)**
 for PR 11178 at commit 
[`31854eb`](https://github.com/apache/spark/commit/31854eb8a35a405c345d108ab474576523a8de3a).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185320485
  
**[Test build #51435 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51435/consoleFull)**
 for PR 11178 at commit 
[`31854eb`](https://github.com/apache/spark/commit/31854eb8a35a405c345d108ab474576523a8de3a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185311768
  
@ScrapCodes, it's some transient Jenkins flakiness, not caused by this PR>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185311556
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185137373
  
Not sure what is causing this:

```
git fetch --tags --progress https://github.com/apache/spark.git
+refs/pull/11178/*:refs/remotes/origin/pr/11178/* # timeout=15
ERROR: Timeout after 15 minutesERROR
: Error fetching
remote repo 'origin'
```


Prashant Sharma



On Wed, Feb 17, 2016 at 2:07 PM, UCB AMPLab 
wrote:

> Test FAILed.
> Refer to this link for build results (access rights to CI server needed):
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51422/
> Test FAILed.
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185094981
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51422/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185094979
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185090816
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-185015502
  
Looks good !, I have taken a quick look and did not actually ran it. Hoping 
the tests will ensure that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r53119771
  
--- Diff: dev/run-tests.py ---
@@ -336,7 +336,6 @@ def build_spark_sbt(hadoop_version):
 # Enable all of the profiles for the build:
 build_profiles = get_hadoop_profiles(hadoop_version) + 
modules.root.build_profile_flags
 sbt_goals = ["package",
- "assembly/assembly",
--- End diff --

Understood, for tests assembly is no longer needed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r53119446
  
--- Diff: dev/run-tests.py ---
@@ -336,7 +336,6 @@ def build_spark_sbt(hadoop_version):
 # Enable all of the profiles for the build:
 build_profiles = get_hadoop_profiles(hadoop_version) + 
modules.root.build_profile_flags
 sbt_goals = ["package",
- "assembly/assembly",
--- End diff --

See https://issues.apache.org/jira/browse/SPARK-9284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/11178#discussion_r53119369
  
--- Diff: dev/run-tests.py ---
@@ -336,7 +336,6 @@ def build_spark_sbt(hadoop_version):
 # Enable all of the profiles for the build:
 build_profiles = get_hadoop_profiles(hadoop_version) + 
modules.root.build_profile_flags
 sbt_goals = ["package",
- "assembly/assembly",
--- End diff --

A full assembly is no longer needed ?, how do you configure classpath ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-184993594
  
@JoshRosen I am sorry for the delay here, I will try to do it today itself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-184987533
  
Jenkins, retest this please.

Any comments here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183613871
  
This patch ended up changing substantially, so I'd like @ScrapCodes to take 
a quick look at it.

In a nutshell:

- I modified the `tools` project to remove a direct dependency on Spark, so 
`sbt/sbt tools/fullClasspath` will now return the classpath for the 
`GenerateMIMAIgnore` class itself plus its own dependencies.
   - This required me to delete two classes full of dead code that we don't 
use anymore
- `GenerateMIMAIgnore` now uses 
[ClassUtil](http://software.clapper.org/classutil/) to find all of the Spark 
classes rather than our homemade JAR traversal code. The problem in our own 
code was that it didn't handle folders of classes properly, which is necessary 
in order to generate excludes with an assembly-free Spark build.
- `./dev/mima` no longer runs through `spark-class`, eliminating the need 
to reason about classpath ordering between `SPARK_CLASSPATH` and the assembly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183588756
  
**[Test build #51228 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51228/consoleFull)**
 for PR 11178 at commit 
[`31854eb`](https://github.com/apache/spark/commit/31854eb8a35a405c345d108ab474576523a8de3a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183610345
  
**[Test build #51228 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51228/consoleFull)**
 for PR 11178 at commit 
[`31854eb`](https://github.com/apache/spark/commit/31854eb8a35a405c345d108ab474576523a8de3a).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183610661
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183610663
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51228/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183526285
  
**[Test build #51214 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51214/consoleFull)**
 for PR 11178 at commit 
[`76a365e`](https://github.com/apache/spark/commit/76a365eee6dddb975ec9acc4c0ffaa562bfb6c63).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183545194
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51214/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183545098
  
**[Test build #51214 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51214/consoleFull)**
 for PR 11178 at commit 
[`76a365e`](https://github.com/apache/spark/commit/76a365eee6dddb975ec9acc4c0ffaa562bfb6c63).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183545191
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183575329
  
**[Test build #51224 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51224/consoleFull)**
 for PR 11178 at commit 
[`9fc0f7a`](https://github.com/apache/spark/commit/9fc0f7acd2c3d6dd8ccf158affa5ef12c8d56072).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183575382
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183575387
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51224/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183570979
  
**[Test build #51224 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51224/consoleFull)**
 for PR 11178 at commit 
[`9fc0f7a`](https://github.com/apache/spark/commit/9fc0f7acd2c3d6dd8ccf158affa5ef12c8d56072).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183206102
  
**[Test build #51176 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51176/consoleFull)**
 for PR 11178 at commit 
[`bef62eb`](https://github.com/apache/spark/commit/bef62ebb8ec5065061ff0ca49a4cb7e0182c47b6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183211270
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183211245
  
**[Test build #51176 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51176/consoleFull)**
 for PR 11178 at commit 
[`bef62eb`](https://github.com/apache/spark/commit/bef62ebb8ec5065061ff0ca49a4cb7e0182c47b6).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183211271
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51176/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/11178

[SPARK-13294] [PROJECT INFRA] Don't build full assembly in dev/run-tests

As of SPARK-9284 we should no longer need to build the full Spark assembly 
JAR in order to run tests. Therefore, we should remove the assembly step from 
{{dev/run-tests}} in order to reduce build + test time.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark remove-assembly-in-run-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11178.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11178


commit 73b15504152e4ca3f82a120d255358f12cf360a4
Author: Josh Rosen 
Date:   2016-02-12T01:19:33Z

[BUILD] Remove assembly/assembly step from dev/run-tests.

commit b6f1ce89dd4e65885abca9d22665aa518eebf10b
Author: Josh Rosen 
Date:   2016-02-12T01:21:44Z

Also remove from Maven build step.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183146095
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183146096
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51162/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183150046
  
LGTM provided tests pass.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183143279
  
**[Test build #51162 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51162/consoleFull)**
 for PR 11178 at commit 
[`b6f1ce8`](https://github.com/apache/spark/commit/b6f1ce89dd4e65885abca9d22665aa518eebf10b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183146069
  
**[Test build #51162 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51162/consoleFull)**
 for PR 11178 at commit 
[`b6f1ce8`](https://github.com/apache/spark/commit/b6f1ce89dd4e65885abca9d22665aa518eebf10b).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183179580
  
**[Test build #51168 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51168/consoleFull)**
 for PR 11178 at commit 
[`5528c48`](https://github.com/apache/spark/commit/5528c48a7524952d3cc1f2d2a2bd303696c07f59).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183181036
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51168/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183181035
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13294] [PROJECT INFRA] Don't build full...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11178#issuecomment-183181025
  
**[Test build #51168 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51168/consoleFull)**
 for PR 11178 at commit 
[`5528c48`](https://github.com/apache/spark/commit/5528c48a7524952d3cc1f2d2a2bd303696c07f59).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org